Files
Jungfraujoch/.gitea/workflows/build_and_test.yml
T
leonarski_fandFilip Leonarski 19b135186d
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 20m49s
Build Packages / Create release (push) Successful in 7s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 4m56s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m10s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 13m51s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 16m56s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 20m59s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 21m25s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 21m17s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 14m14s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 25m59s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 25m48s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 15m52s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 19m33s
Build Packages / Generate python client (push) Successful in 29s
Build Packages / Build documentation (push) Successful in 1m19s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 25m31s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 25m24s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 26m5s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 22m19s
Build Packages / Unit tests (push) Failing after 2h28m24s
Build Packages / Validation Battery (push) Successful in 40m33s
v1.0.0-rc.175 (#85)
* Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`).
* Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection.
* Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS).
* Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead.
* Rugnux: On macOS, Bragg integration runs on the Apple GPU.
* Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand.
* Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`.
* Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit.
* Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed.
* Rugnux: The beam-stop shadow is found correctly on tilted detectors.
* Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement.
* jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps.
* jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection.
* jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images.
* Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges.

---------

Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home>
Reviewed-on: #85
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-10-16 15:34:43 +02:00

996 lines
51 KiB
YAML

name: Build Packages
env:
# Architecture and link-time optimisation for every Linux configure. Both are deliberately absent
# from CMakeLists.txt so a site can pick its own (x86-64-v4 on an AVX-512 cluster, -march=native,
# or the plain baseline); CI pins what it ships.
#
# -flto=auto needs GCC >= 10 and both CI images put gcc-toolset-13 on PATH. Measured on rugnux
# against an otherwise identical build: 7-10% fewer retired instructions and a 9% smaller binary,
# but only ~1.5% off the wall clock, because the offline pipeline is GPU- and I/O-bound rather
# than CPU-instruction-bound. It costs about 3x on an INCREMENTAL rebuild (one file plus link,
# 9.8 s -> 30.1 s), which is why it belongs here and not in CMakeLists: CI always builds from
# scratch and ships the result, so it pays the link once and a developer never pays it at all.
#
# Not CMAKE_INTERPROCEDURAL_OPTIMIZATION, which is the tidier spelling and would cover MSVC too:
# built and linked fine (CUDA included, and it does not reach .cu either way), but measured 3%
# MORE retired instructions than -flto=auto - 396.9 G vs 384.9 G over three runs each against a
# 0.45% run-to-run spread, so the gap is real. It also leaves two dependencies (FFTW, libzmq) out
# of LTO that the flag reaches, and it did not switch CMAKE_AR to gcc-ar here, so the one
# correctness argument for it did not materialise either.
LINUX_CMAKE_FLAGS: '-DCMAKE_CXX_FLAGS="-march=x86-64-v3 -flto=auto" -DCMAKE_C_FLAGS="-march=x86-64-v3 -flto=auto"'
# MSVC has no spelling for the x86-64-v2 level; /arch:AVX is the nearest and implies SSE4.1/4.2,
# which is what matters here - without it Eigen has no vectorised round and falls back to a libm
# call per element. AVX is Sandy Bridge (2011) and up, a safe floor for a desktop viewer.
# Setting CMAKE_CXX_FLAGS replaces CMake's MSVC default ("/DWIN32 /D_WINDOWS /GR /EHsc"), so
# CMakeLists.txt adds /EHsc back itself.
MSVC_ARCH_CMAKE_FLAGS: '-DCMAKE_CXX_FLAGS="/arch:AVX" -DCMAKE_C_FLAGS="/arch:AVX"'
on:
push:
branches:
- '**'
# No pull_request trigger: branches live in this repository, so the push above has already run
# the whole workflow on the very same commit. Adding the PR event only ran it a second time.
workflow_dispatch:
inputs:
create_release:
type: boolean
description: 'Create release'
required: false
default: false
battery_subset:
description: 'Validation battery: "--tier ci" (default), "--tier smoke", "--only 5reo,lyso_x06da_ref", or empty for the full battery (open + inhouse). Not run with a release.'
required: false
default: '--tier ci'
battery_label:
description: 'Validation battery run label (part of the run directory name)'
required: false
default: 'manual'
jobs:
# ---- Portable Linux artifacts: viewer, rugnux, XDS plugin -----------------------------------------
# One viewer-only build per variant packs three tarballs, one program each (see CPack in
# CMakeLists.txt): the viewer build compiles everything rugnux and the plugin need anyway.
build-portable-tgz:
name: build:portable:linux-x86_64:${{ matrix.variant }}
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
include:
- variant: cuda
use_cuda: 'ON'
- variant: nocuda
use_cuda: 'OFF'
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build # self-hosted runners reuse the workspace volume; start from a clean build dir
- name: Configure
shell: bash
run: |
mkdir -p build
cd build
cmake -G Ninja -DJFJOCH_VIEWER_ONLY=ON -DJFJOCH_USE_CUDA=${{ matrix.use_cuda }} ${{ env.LINUX_CMAKE_FLAGS }} -DCMAKE_BUILD_TYPE=Release ..
- name: Build viewer, rugnux and XDS plugin tgz
shell: bash
run: |
cd build
ninja -j16
cpack
- name: Verify one libjpeg in the viewer
shell: bash
run: |
set -euo pipefail
# With the static Qt, Qt's JPEG plugin brings its own libjpeg-turbo (another ABI) and the
# linker silently mixes it with ours (see viewer/CMakeLists.txt). Files, not pipes into
# `grep -q`, for the SIGPIPE reason given in the rugnux check below.
strings build/viewer/jfjoch_viewer > /tmp/jfjv.strings
if grep -qE 'QJpegHandler|QTiffHandler' /tmp/jfjv.strings; then echo "a Qt JPEG/TIFF plugin is linked"; exit 1; fi
grep -c 'libjpeg-turbo version' /tmp/jfjv.strings > /tmp/jfjv.njpeg || true
if [ "$(cat /tmp/jfjv.njpeg)" != 1 ]; then echo "expected one libjpeg-turbo, found $(cat /tmp/jfjv.njpeg)"; exit 1; fi
echo "one libjpeg-turbo, no Qt JPEG/TIFF plugin"
- name: Verify rugnux is self-contained
shell: bash
run: |
set -euo pipefail
rm -rf /tmp/rgx && mkdir -p /tmp/rgx
tar xf build/rugnux-*.tar.gz -C /tmp/rgx
bin=$(find /tmp/rgx -name rugnux -type f)
# Each producer writes to a file and the greps read files, rather than piping into
# `grep -q`: grep exits at the first match, the producer then dies of SIGPIPE, and under
# `set -o pipefail` that becomes the pipeline's status -- which a leading `!` would turn
# into a pass, hiding the very thing being tested.
ldd "$bin" > /tmp/rgx.ldd 2>&1 || true
if grep -q cufft /tmp/rgx.ldd; then echo "cuFFT is linked dynamically"; exit 1; fi
find /tmp/rgx -name 'libcufft*' > /tmp/rgx.cufft
if [ -s /tmp/rgx.cufft ]; then echo "a libcufft is shipped in the tarball"; exit 1; fi
echo "self-contained: static cuFFT, nothing to ship beside it"
- name: Verify the XDS plugin exports
shell: bash
run: |
set -euo pipefail
rm -rf /tmp/xdsp && mkdir -p /tmp/xdsp
tar xf build/jfjoch_xds_plugin-*.tar.gz -C /tmp/xdsp
so=$(find /tmp/xdsp -name 'libjfjoch_xds_plugin.so.*' -type f)
# only the four plugin_* entry points are exported
nm -D --defined-only "$so" | awk '{print $3}' > /tmp/xdsp.exports
if grep -v '^plugin_' /tmp/xdsp.exports; then echo "the plugin exports more than plugin_*"; exit 1; fi
ls /tmp/xdsp/share/doc/jfjoch_xds_plugin/licenses
echo "exports plugin_* only"
- name: Upload tgz to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
shopt -s nullglob
# CMakeLists.txt names the archives jfjoch_viewer-<version>-linux-{cuda<major>|cpu},
# rugnux-<version>-linux-x86_64-{cuda<major>|cpu} and jfjoch_xds_plugin-<version>-linux-x86_64
# (.tar.gz); rename to .tgz before upload. The plugin does not use CUDA and comes out the
# same from both variants, so only one of them uploads it.
files=(build/jfjoch_viewer-*.tar.gz build/rugnux-*.tar.gz)
if [ "${{ matrix.variant }}" = "nocuda" ]; then
files+=(build/jfjoch_xds_plugin-*.tar.gz)
fi
if [ ${#files[@]} -lt 2 ]; then
echo "Archives missing in build/"
exit 1
fi
for file in "${files[@]}"; do
tgz="${file%.tar.gz}.tgz"
mv "$file" "$tgz"
python3 gitea_upload_file.py "$tgz"
done
# One build packs the viewer installer and the rugnux .zip (CPackProjectConfig.cmake in
# CMakeLists.txt gives each generator its components).
build-windows:
name: build:portable:windows-x86_64:${{ matrix.variant }}
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: windows-11-cuda-qt
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
include:
- variant: cuda
use_cuda: 'ON'
- variant: nocuda
use_cuda: 'OFF'
steps:
- uses: actions/checkout@v4
- name: Configure viewer build
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cmake -G Ninja -B build -DJFJOCH_USE_CUDA=${{ matrix.use_cuda }} -DCMAKE_BUILD_TYPE=Release ${{ env.MSVC_ARCH_CMAKE_FLAGS }} -DCMAKE_PREFIX_PATH="C:/deps;C:/Qt/6.11.1/msvc2022_64"
- name: Build viewer
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cmake --build build
# The quick cross-platform checks (tests/PortableTest.cpp and the [portable] cases), before
# anything is packaged. CPU-only: the runner has no GPU.
- name: Build portable tests
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cmake --build build --target jfjoch_portable_test
- name: Run portable tests
shell: cmd
run: |
cd build\tests
jfjoch_portable_test.exe "[portable]"
- name: Build installer (NSIS) and rugnux zip
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cd build
cpack
- name: Verify the rugnux zip
shell: powershell
run: |
# rugnux.exe and, in the CUDA variant, the cuFFT DLL it needs - and no viewer.
$zip = Get-ChildItem -Path build -Filter 'rugnux-*-win64-*.zip'
if ($zip.Count -ne 1) { throw 'Expected one rugnux zip in build/' }
Add-Type -AssemblyName System.IO.Compression.FileSystem
$names = [System.IO.Compression.ZipFile]::OpenRead($zip[0].FullName).Entries | ForEach-Object { $_.FullName }
$names
if (-not ($names -match 'rugnux\.exe$')) { throw 'rugnux.exe missing from the zip' }
if ($names -match 'jfjoch_viewer') { throw 'the viewer is in the rugnux zip' }
if ('${{ matrix.use_cuda }}' -eq 'ON' -and -not ($names -match 'cufft64_.*\.dll$')) { throw 'cuFFT DLL missing from the CUDA zip' }
- name: Upload installer and rugnux zip to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: powershell
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
# NSIS installer jfjoch-viewer-<version>-win64-{cuda<major>|cpu}.exe and
# rugnux-<version>-win64-{cuda<major>|cpu}.zip (see CMakeLists.txt).
$files = @(Get-ChildItem -Path build -Filter 'jfjoch-viewer-*-win64-*.exe') + @(Get-ChildItem -Path build -Filter 'rugnux-*-win64-*.zip')
if ($files.Count -ne 2) { throw 'Expected the installer and the rugnux zip in build/' }
foreach ($file in $files) {
python gitea_upload_file.py $file.FullName
if ($LASTEXITCODE -ne 0) { throw "Upload failed for $($file.Name)" }
}
build-macos:
name: build:portable:macos-arm64:nocuda
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
# A Mac mini running act_runner in host mode (no containers on macOS): the job runs as the runner's
# user with that user's tools - Xcode command line tools, CMake.app and Qt under ~/Qt - so the
# paths below are this machine's, like the Windows runner's C:/Qt.
runs-on: macos-arm64
timeout-minutes: 120
env:
QT_PREFIX: /Users/jungfrau/Qt/6.11.2/macos
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build-macos # host runners keep the workspace between runs
# The Developer ID in the runner's login keychain signs the app (hardened runtime, secure
# timestamp) so the .dmg can be notarized below; without it the build must not go on, as an
# ad-hoc signed download is refused by Gatekeeper. Passed on as the certificate's SHA-1 (see
# JFJOCH_MACOS_SIGNING_IDENTITY in CMakeLists.txt). The notarytool credentials sit in the same
# keychain under the profile jfjoch-notary (xcrun notarytool store-credentials --keychain
# ~/Library/Keychains/login.keychain-db: that file never locks here, the default data-protection
# keychain does with the screen).
- name: Signing identity
shell: bash
run: |
set -euo pipefail
identity=$(security find-identity -v -p codesigning | sed -n 's/^ *[0-9]*) \([0-9A-F]*\) "Developer ID Application: .*/\1/p' | head -1)
if [ -z "$identity" ]; then echo "no Developer ID Application identity in the runner's keychain"; exit 1; fi
echo "MACOS_SIGNING_IDENTITY=$identity" >> "$GITHUB_ENV"
- name: Configure
shell: bash
# JFJOCH_VIEWER_ONLY is forced on macOS. No -mcpu: the arm64 baseline is what every Apple
# Silicon Mac runs, and NEON is part of it.
run: cmake -S . -B build-macos -DCMAKE_BUILD_TYPE=Release -DJFJOCH_USE_CUDA=OFF -DCMAKE_PREFIX_PATH="$QT_PREFIX" -DJFJOCH_MACOS_SIGNING_IDENTITY="$MACOS_SIGNING_IDENTITY"
# The quick cross-platform checks (tests/PortableTest.cpp and the [portable] cases), before
# anything is packaged.
- name: Portable tests
shell: bash
run: |
cmake --build build-macos --target jfjoch_portable_test -j "$(sysctl -n hw.ncpu)"
cd build-macos/tests && ./jfjoch_portable_test "[portable]~[Metal]"
# The Metal Bragg integration engine has to give the CPU engine's result bit for bit
# (BraggIntegrationEngineMetal.h): every reflection's every float, on random crowded frames
# through every integrator, overlap and background mode. Its own step, so a mismatch is named as
# such; it fails rather than skips on a machine without a Metal device, as this is the only job
# that can check it.
- name: Metal integration engine is bit-identical to the CPU engine
shell: bash
run: cd build-macos/tests && ./jfjoch_portable_test "[Metal]"
# One build, two artifacts: the viewer .dmg and rugnux as .tar.gz (CPACK_PROJECT_CONFIG_FILE
# in CMakeLists.txt gives each generator its components). The XDS plugin is Linux-only.
- name: Build viewer DMG and rugnux tgz
shell: bash
run: |
cmake --build build-macos -j "$(sysctl -n hw.ncpu)"
cd build-macos && cpack
- name: Verify the bundle is self-contained
shell: bash
run: |
set -euo pipefail
app=$(ls -d build-macos/_CPack_Packages/Darwin/DragNDrop/*/ALL_IN_ONE/jfjoch_viewer.app)
# The frameworks sit in the bundle; the executable must look for them there ...
otool -l "$app/Contents/MacOS/jfjoch_viewer" > /tmp/jfjv.load
if ! grep -q "@executable_path/../Frameworks" /tmp/jfjv.load; then echo "no rpath into the bundle"; exit 1; fi
# ... and nothing in the bundle may point at this machine's Qt (or anything else under /Users)
find "$app" -type f \( -perm +111 -o -name '*.dylib' \) -exec otool -L {} + > /tmp/jfjv.libs 2>/dev/null || true
if grep -q "/Users/" /tmp/jfjv.libs; then grep "/Users/" /tmp/jfjv.libs; echo "absolute path into the build machine"; exit 1; fi
# the Developer ID signature must survive everything installed into the bundle, and carry
# the hardened runtime that notarization demands
codesign --verify --deep --strict "$app"
# rugnux is in an archive of its own, not in the app
find "$app" -name rugnux > /tmp/jfjv.extra
if [ -s /tmp/jfjv.extra ]; then cat /tmp/jfjv.extra; echo "the bundle carries another program"; exit 1; fi
# to a file, not piped into `grep -q` (see "Verify rugnux is self-contained" above)
codesign -d --verbose=2 "$app" > /tmp/jfjv.sig 2>&1
if ! grep -q "flags=.*runtime" /tmp/jfjv.sig; then cat /tmp/jfjv.sig; echo "no hardened runtime"; exit 1; fi
echo "self-contained and validly signed"
- name: Verify rugnux is self-contained
shell: bash
run: |
set -euo pipefail
rm -rf /tmp/rgxm && mkdir -p /tmp/rgxm
tar xf build-macos/rugnux-*.tar.gz -C /tmp/rgxm
bin=$(find /tmp/rgxm -name rugnux -type f)
# Only the system's own libraries may be linked dynamically; everything else is static.
otool -L "$bin" | tail -n +2 > /tmp/rgxm.libs
if grep -v -e "/usr/lib/" -e "/System/Library/" /tmp/rgxm.libs; then echo "links a non-system library"; exit 1; fi
# and it runs: with no arguments rugnux prints its usage
"$bin" > /tmp/rgxm.out 2>&1 || true
if ! grep -qi "usage" /tmp/rgxm.out; then cat /tmp/rgxm.out; echo "rugnux did not start"; exit 1; fi
# and carries the Developer ID signature with the hardened runtime that notarization demands
codesign --verify --strict "$bin"
codesign -d --verbose=2 "$bin" > /tmp/rgxm.sig 2>&1
if ! grep -q "flags=.*runtime" /tmp/rgxm.sig; then cat /tmp/rgxm.sig; echo "no hardened runtime"; exit 1; fi
echo "self-contained, runs and is validly signed"
# Every push, not only a release: a rejection by Apple (a library macdeployqt left unsigned, an
# entitlement the hardened runtime needs) should show up when the change is made, not when the
# release is cut. Signs the image, waits for Apple's verdict, staples the ticket to the image;
# the rugnux binary only gets its hash registered, a bare executable has nowhere to staple.
- name: Notarize
shell: bash
run: tools/notarize_macos.sh build-macos/jfjoch-viewer-*.dmg build-macos/rugnux-*.tar.gz
- name: Upload DMG and tgz to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
shopt -s nullglob
# CMakeLists.txt names them jfjoch-viewer-<version>-macos-arm64.dmg and
# rugnux-<version>-macos-arm64-cpu.tar.gz; the archive is renamed to .tgz before upload,
# as the Linux ones are.
dmgs=(build-macos/jfjoch-viewer-*.dmg)
tars=(build-macos/rugnux-*.tar.gz)
if [ ${#dmgs[@]} -ne 1 ] || [ ${#tars[@]} -ne 1 ]; then
echo "Expected one DMG and one archive in build-macos/"
exit 1
fi
python3 gitea_upload_file.py "${dmgs[0]}"
for file in "${tars[@]}"; do
tgz="${file%.tar.gz}.tgz"
mv "$file" "$tgz"
python3 gitea_upload_file.py "$tgz"
done
# ---- Rugnux --------------------------------------------------------------------------------------
build-rugnux-arm:
name: build:rugnux:linux-aarch64:cuda # cross-compiled on x86_64
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
# Cross-compiled on an x86_64 runner: nvcc, gcc and cmake all run native and only emit aarch64,
# so this costs about what a normal build costs. Uses the ordinary ubuntu2404 image, which now
# also carries the aarch64 toolchain and the CUDA sbsa cross tree.
#
# The toolchain file comes from the CHECKOUT, not from the image. It describes how to build this
# source, so it belongs with the source: baked into the image it could only be changed by
# rebuilding and re-pushing the image, and a fix to it would appear to have no effect.
runs-on: jfjoch_ubuntu2404
container:
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2404:2609
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build-aarch64 # self-hosted runners reuse the workspace volume
- name: Configure (cross to aarch64 SBSA)
shell: bash
# NOT ${{ env.LINUX_CMAKE_FLAGS }}: that pins -march=x86-64-v3, which an aarch64 compiler
# rejects. The aarch64 baseline already has NEON and a vectorised round, so the Eigen gap
# that flag closes on x86 does not exist here.
# sm_90 = GH200 (Grace Hopper, e.g. ALPS), sm_121 = GB10 (DGX Spark). One tarball, both.
run: |
cmake -G Ninja -S . -B build-aarch64 \
-DCMAKE_TOOLCHAIN_FILE="$PWD/docker/ubuntu2404/aarch64-sbsa.cmake" \
-DCMAKE_BUILD_TYPE=Release \
-DJFJOCH_RUGNUX_ONLY=ON \
-DJFJOCH_USE_CUDA=ON \
-DJFJOCH_CUDA_ARCHITECTURES="90;121"
- name: Build and package
shell: bash
run: cmake --build build-aarch64 --target package -j16
# A cross build that silently produced x86_64 still packages cleanly, so assert the artifact.
- name: Verify the tarball is really aarch64
shell: bash
run: |
set -euo pipefail
tgz=$(ls build-aarch64/rugnux-*.tar.gz | head -1)
echo "checking $tgz"
rm -rf /tmp/tgzcheck && mkdir -p /tmp/tgzcheck
tar xf "$tgz" -C /tmp/tgzcheck
bin=$(find /tmp/tgzcheck -name rugnux -type f | head -1)
# Everything is captured to a file first and the greps read files. Piping a large
# producer into `grep -q` makes it die of SIGPIPE once grep exits, which `set -o pipefail`
# turns into a failure -- and on a negated check into a spurious PASS.
file "$bin" > /tmp/tc.bin
cat /tmp/tc.bin
if ! grep -q "ARM aarch64" /tmp/tc.bin; then echo "not an aarch64 binary"; exit 1; fi
# nothing x86 may have leaked in (libjpeg-turbo's ExternalProject is the classic culprit)
find /tmp/tgzcheck -type f -exec file {} + > /tmp/tc.arch
if grep -q "x86-64" /tmp/tc.arch; then echo "an x86-64 file leaked into the tarball"; exit 1; fi
# cuFFT must be linked statically: nothing to ship beside the binary, nothing to find
ldd "$bin" > /tmp/tc.ldd 2>&1 || true
if grep -q cufft /tmp/tc.ldd; then echo "cuFFT is linked dynamically"; exit 1; fi
# both GPU targets present in the fatbin: sm_90 = GH200, sm_121 = GB10
cuobjdump --list-elf "$bin" > /tmp/tc.elf
for arch in sm_90 sm_121; do
if ! grep -q "$arch" /tmp/tc.elf; then echo "fatbin is missing $arch"; exit 1; fi
done
echo "aarch64, self-contained, both GPU targets present"
- name: Upload tarball to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
shopt -s nullglob
files=(build-aarch64/rugnux-*.tar.gz)
if [ ${#files[@]} -eq 0 ]; then
echo "No rugnux archive found in build-aarch64/"
exit 1
fi
for file in "${files[@]}"; do
tgz="${file%.tar.gz}.tgz"
mv "$file" "$tgz"
python3 gitea_upload_file.py "$tgz"
done
# ---- Jungfraujoch --------------------------------------------------------------------------------
unit-tests:
name: Unit tests
# A release built by hand does not re-run the suite: the push that put the commit on a branch
# already ran it on that very commit, and so does the merge to main. Nothing is skipped that
# was not already run, and a release build is the one case where the wait is pure delay.
if: "!(github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true))"
runs-on: jfjoch_rocky8
# The validation battery below runs on the same machine and GPU, and its rugnux takes the card's
# memory as its sets need it: a GPU test beside it finds the card full, and a GPU out-of-memory
# ends the run by design. The battery waits for this job (needs:), so the two take turns.
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
options: --gpus all
timeout-minutes: 90
env:
CTEST_OUTPUT_ON_FAILURE: '1'
steps:
- uses: actions/checkout@v4
- name: Configure auth and fetch LFS
shell: bash
env:
GITEA_TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
git lfs install --local
AUTH=$(git config --local http.${{ github.server_url }}/.extraheader)
git config --local --unset http.${{ github.server_url }}/.extraheader
git config --local http.${{ github.server_url }}/${{ github.repository }}.git/info/lfs/objects/batch.extraheader "$AUTH"
git lfs pull
- name: Build tests
shell: bash
run: |
mkdir -p build
cd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release ${{ env.LINUX_CMAKE_FLAGS }} ..
ninja -j16 jfjoch_test jfjoch_hdf5_test jfjoch_hdf5_enospc_test enospc_shim
- name: Run unit tests
shell: bash
run: |
cd build/tests
./jfjoch_test
- name: HDF5 ENOSPC test
shell: bash
run: |
cd build/tests
LD_PRELOAD=./enospc_shim.so ./jfjoch_hdf5_enospc_test HDF5File_enospc
LD_PRELOAD=./enospc_shim.so ./jfjoch_hdf5_enospc_test FileWriter_enospc
- name: Run hdf5 test
shell: bash
run: |
cd build/tools
./jfjoch_hdf5_test ../../tests/test_data/compression_benchmark.h5
# ---- Validation battery ------------------------------------------------------------------------
# Every push runs the ci tier of tools/battery (29 sets, about 11 minutes of rugnux on an RTX 5080,
# plus the build); the full battery (open + inhouse arms, about 2 hours of rugnux there) runs only
# when started by hand (Run workflow, battery_subset empty; no release). It runs on the one machine
# that holds the battery data, after the unit tests of the same run (needs:), which share its GPU,
# under its own runner label, with the data mounted read-only and /data/battery (scratch, kept
# reports, PDB/COD caches, the full-run lock) read-write. No concurrency group: a group keeps one
# pending job and cancels it for a newer one, so a merge to main could lose its battery to a
# branch push. Two pushes close together can still overlap on the GPU. The site
# config below names only the open and inhouse arms; the private arm is never run here.
#
# A ci-tier run is followed by the wrong-model null, as a step of its own with its own pass/fail
# (battery.py nullmodel; tools/battery/nullmodel.json: five sets, each without --model and with five
# wrong models - other open-arm entries' deposited coordinates, from the PDB cache under
# /data/battery - 30 more rugnux runs, about 6 minutes), in a run directory and report of its own, so
# the ci-tier rows are those of a run without it.
#
# The battery step fails when a ci-tier set fails on lattice or space group, crashes or has no input -
# in any run that includes those sets, so a manual full run fails on them too. The wrong-model step
# fails when a pair fails (the data or a decision moved, or the wrong model was not recognised as
# wrong). Each step runs whatever the other did, so the job page says which of the two failed. Every
# other failure and every metric is reported, not judged: read the table at the end of each step and
# the reports. Each run's report.md, results.json and manifest.json are kept, in
# /data/battery/ci_reports/<run>/ and (with the log) as the job's artifact - what `battery.py keep`
# needs to commit a run under tools/battery/results/; the run directories themselves are scratch.
#
# Runner side (/opt/gitea/config.yaml on that machine, then restart the runner):
# runner.labels: + "jfjoch_battery:docker://gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609"
# runner.timeout: 12h (the default 3h kills a full battery)
# container.valid_volumes: ["/data/open", "/data/inhouse", "/data/battery", "/opt/xtal"]
# CCP4 (SHELXL for the small-molecule check, REFMAC for --model-check) installed at /opt/xtal/ccp4-9
battery:
name: Validation Battery
# After the unit tests, which share the machine's GPU; whatever their outcome, so a failing unit
# test does not hide the battery's result. Not with a release, as the unit tests.
needs: unit-tests
if: "!cancelled() && !(github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true))"
runs-on: jfjoch_battery
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
# nvidia-smi inside the container still counts every process on the GPU (host and other
# containers), so the battery's gpu_others / "timing NOT a reference" marking keeps working.
options: --gpus all
volumes:
- /data/open:/data/open:ro
- /data/inhouse:/data/inhouse:ro
- /data/battery:/data/battery
- /opt/xtal:/opt/xtal:ro
timeout-minutes: 720
env:
# A push runs the ci tier; a manual run takes what it was given (empty = full battery).
SUBSET: ${{ github.event_name == 'push' && '--tier ci' || github.event.inputs.battery_subset }}
LABEL: ${{ github.event_name == 'push' && 'ci' || github.event.inputs.battery_label }}
steps:
- uses: actions/checkout@v4
- name: Build rugnux
id: build
shell: bash
run: |
rm -rf build && mkdir build && cd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release ${{ env.LINUX_CMAKE_FLAGS }} ..
ninja -j16 rugnux
- name: Python for the scorer
shell: bash
run: |
# the image's python3 is 3.6, too old for gemmi
dnf -y -q install python3.12 python3.12-pip
python3.12 -m venv /tmp/bvenv
/tmp/bvenv/bin/pip install -q gemmi h5py
- name: Site config
shell: bash
run: |
# The run directory (merged files, logs, results.json) is scratch, deleted after the job;
# only report.md is kept, in /data/battery/ci_reports and as the job's artifact.
cat > /tmp/site.json <<EOF
{
"runs_root": "/data/battery/scratch/${{ github.run_id }}",
"lock_file": "/data/battery/full_run.lock",
"gpulock": null,
"baseline": null,
"baseline_private": null,
"pdb_cache": "/data/battery/pdb_cache",
"cod_cache": "/data/battery/cod_cache",
"shelxl": "/opt/xtal/ccp4-9/bin/shelxl",
"arms": {
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/data/open"},
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/data/inhouse"}
}
}
EOF
cat /tmp/site.json
- name: Run the battery
shell: bash
run: |
set -o pipefail
# ends with one line per set (rugnux / reference, a tick or cross on the space group);
# exits non-zero when a ci-tier set failed. The wrong-model null is the next step.
JFJOCH_BATTERY_SITE=/tmp/site.json /tmp/bvenv/bin/python tools/battery/battery.py run \
--rugnux build/rugnux/rugnux --build-dir build --label "$LABEL" --no-nullmodel \
$SUBSET 2>&1 | tee battery.log
- name: Wrong-model null
if: ${{ !cancelled() && steps.build.outcome == 'success' && contains(env.SUBSET, '--tier ci') }}
shell: bash
run: |
set -o pipefail
# one line per pair; exits non-zero when a pair failed
JFJOCH_BATTERY_SITE=/tmp/site.json /tmp/bvenv/bin/python tools/battery/battery.py nullmodel \
--rugnux build/rugnux/rugnux --build-dir build --label "$LABEL-null" 2>&1 | tee nullmodel.log
- name: Keep the report
if: always()
shell: bash
run: |
mkdir -p battery_out /data/battery/ci_reports
missing=0
for log in battery.log nullmodel.log; do
[ -f "$log" ] || continue
run=$(sed -n 's/^run directory //p' "$log" | head -1)
[ -n "$run" ] && [ -f "$run/report.md" ] || { echo "no report for $log"; missing=1; continue; }
# report.md, results.json and manifest.json as the battery wrote them: what
# `battery.py keep` commits under tools/battery/results/<host>/<run>/
name=$(basename "$run")
mkdir -p "battery_out/$name" "/data/battery/ci_reports/$name"
for f in report.md results.json manifest.json; do
[ -f "$run/$f" ] && cp "$run/$f" "battery_out/$name/" && cp "$run/$f" "/data/battery/ci_reports/$name/"
done
cp "$log" "battery_out/$name/"
# the report's own per-arm summary and failures, or the wrong-model table, into the job log
awk '/^## Summary/{p=1} /^### By population/{p=0} p' "$run/report.md"
awk '/^### Failures/{p=1;print;next} /^### /{p=0} p' "$run/report.md"
awk '/^## Wrong-model null/{p=1} /^## Summary/{p=0} p' "$run/report.md"
done
exit $missing
- name: Upload report
if: always()
uses: actions/upload-artifact@v3
with:
name: battery-report
path: battery_out/
- name: Delete the run directory
if: always()
shell: bash
run: |
# a finished run directory is read-only; the container runs as root, which deletes it anyway
rm -rf "/data/battery/scratch/${{ github.run_id }}"
# files written by root in the container go back to the owner of the battery tree
chown -R "$(stat -c %u:%g /data/battery)" /data/battery/ci_reports /data/battery/pdb_cache /data/battery/cod_cache 2>/dev/null || true
build-rpm:
name: build:jfjoch:${{ matrix.platform }}:${{ matrix.variant }}
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: ${{ matrix.runner }}
container:
image: ${{ matrix.image }}
strategy:
fail-fast: false
matrix:
include:
- runner: jfjoch_rocky8
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
distro: rocky8
platform: rocky8
variant: cuda
cmake_flags: -DJFJOCH_USE_CUDA=ON
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el8/slsdet8-cuda12/upload
- runner: jfjoch_rocky8
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
distro: rocky8_sls9
platform: rocky8
variant: cuda-sls9
cmake_flags: -DJFJOCH_USE_CUDA=ON -DSLS9=ON
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el8/slsdet9-cuda12/upload
- runner: jfjoch_rocky8
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
distro: rocky8_nocuda
platform: rocky8
variant: nocuda
cmake_flags: -DJFJOCH_USE_CUDA=OFF
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el8/slsdet8-nocuda/upload
- runner: jfjoch_rocky9
image: gitea.psi.ch/leonarski_f/jfjoch_rocky9:2609
distro: rocky9
platform: rocky9
variant: cuda
cmake_flags: -DJFJOCH_USE_CUDA=ON
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el9/slsdet8-cuda13/upload
- runner: jfjoch_rocky9
image: gitea.psi.ch/leonarski_f/jfjoch_rocky9:2609
distro: rocky9_sls9
platform: rocky9
variant: cuda-sls9
cmake_flags: -DJFJOCH_USE_CUDA=ON -DSLS9=ON
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el9/slsdet9-cuda13/upload
- runner: jfjoch_rocky9
image: gitea.psi.ch/leonarski_f/jfjoch_rocky9:2609
distro: rocky9_nocuda
platform: rocky9
variant: nocuda
cmake_flags: -DJFJOCH_USE_CUDA=OFF
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el9/slsdet8-nocuda/upload
- runner: jfjoch_ubuntu2204
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2204:2609
distro: ubuntu2204
platform: ubuntu2204
variant: cuda
cmake_flags: -DJFJOCH_USE_CUDA=ON
pkg_glob: "*.deb"
upload_url: https://gitea.psi.ch/api/packages/mx/debian/pool/jammy/cuda13/upload
- runner: jfjoch_ubuntu2204
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2204:2609
distro: ubuntu2204_nocuda
platform: ubuntu2204
variant: nocuda
cmake_flags: -DJFJOCH_USE_CUDA=OFF
pkg_glob: "*.deb"
upload_url: https://gitea.psi.ch/api/packages/mx/debian/pool/jammy/nocuda/upload
- runner: jfjoch_ubuntu2404
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2404:2609
distro: ubuntu2404
platform: ubuntu2404
variant: cuda
cmake_flags: -DJFJOCH_USE_CUDA=ON
pkg_glob: "*.deb"
upload_url: https://gitea.psi.ch/api/packages/mx/debian/pool/noble/cuda13/upload
- runner: jfjoch_ubuntu2404
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2404:2609
distro: ubuntu2404_nocuda
platform: ubuntu2404
variant: nocuda
cmake_flags: -DJFJOCH_USE_CUDA=OFF
pkg_glob: "*.deb"
upload_url: https://gitea.psi.ch/api/packages/mx/debian/pool/noble/nocuda/upload
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build # self-hosted runners reuse the workspace volume; start from a clean build dir
- name: Setup build (cmake)
shell: bash
run: |
mkdir -p build
cd build
cmake -G Ninja -DJFJOCH_INSTALL_DRIVER_SOURCE=ON -DJFJOCH_VIEWER_BUILD=ON ${{ env.LINUX_CMAKE_FLAGS }} -DCMAKE_BUILD_TYPE=Release ${{ matrix.cmake_flags }} ..
- name: Build packages
shell: bash
run: |
cd build
ninja frontend
ninja -j16 package
- name: Upload packages
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
cd build
shopt -s nullglob
files=(${{ matrix.pkg_glob }})
if [ ${#files[@]} -eq 0 ]; then
echo "No package files found for pattern: ${{ matrix.pkg_glob }}"
exit 1
fi
url="${{ matrix.upload_url }}"
if [[ "$url" == *"/rpm/"* ]]; then
url="${url}?sign=true"
fi
for file in "${files[@]}"; do
echo "Uploading $file -> $url"
curl --fail --user __token__:"$TOKEN" --upload-file "$file" "$url"
done
cd ..
# The viewer, rugnux and the XDS plugin are shipped to the release as Linux-version-agnostic
# .tgz by the build-portable-tgz job; only the writer goes to the release as a distro
# package here.
if [ "${{ matrix.distro }}" = "rocky8_nocuda" ]; then
for file in build/jfjoch-writer*rpm; do
python3 gitea_upload_file.py "$file"
done
elif [ "${{ matrix.distro }}" = "rocky9_nocuda" ]; then
for file in build/jfjoch-writer*rpm; do
python3 gitea_upload_file.py "$file"
done
elif [ "${{ matrix.distro }}" = "ubuntu2204_nocuda" ]; then
for file in build/jfjoch*writer*.deb; do
python3 gitea_upload_file.py "$file"
done
fi
hdf5-consumer-tests:
name: HDF5 consumer tests (DIALS, XDS)
runs-on: jfjoch_rocky9
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky9:2609
steps:
- uses: actions/checkout@v4
- name: Build processing test
shell: bash
run: |
mkdir -p build
cd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release ${{ env.LINUX_CMAKE_FLAGS }} ..
ninja -j16 jfjoch_hdf5_test jfjoch_xds_plugin
# One job for what used to be four, each of which configured and built the same
# jfjoch_hdf5_test on the same image. Every case still runs when an earlier one fails -- each
# is a subshell under `set -e`, so a case stops at its own first bad command without taking
# the rest down -- and the summary at the end decides the job's result.
- name: Run every consumer against every file layout
shell: bash
run: |
cd "$GITHUB_WORKSPACE"
HDF5_TEST=$PWD/build/tools/jfjoch_hdf5_test
DATA=$PWD/tests/test_data/compression_benchmark.h5
# Unquoted on use: these are two arguments, not one.
CELL="space_group=P43212 unit_cell=78.551,78.551,36.914,90.000,90.000,90.000"
# -- DIALS -------------------------------------------------------------------------
# dials.import before xia2.ssx, checking the image COUNT: dxtbx reports a short scan
# rather than failing when it cannot reach every image, and xia2.ssx still succeeds on
# the images it did get, so an exit code alone does not notice. Multi-file NXmxLegacy is
# exercised by the XDS cases below but deliberately not here: dxtbx reads only the first
# data file of a legacy master (it takes one dataset out of NXdata and indexes it
# globally), so that combination cannot pass until the limitation is fixed upstream.
dials_case() { # <dir> <prefix> <jfjoch_hdf5_test flags...>
local dir=$1 prefix=$2; shift 2
source /opt/dials-v3-27-0/dials_env.sh
rm -rf "$dir"; mkdir -p "$dir"; cd "$dir"
"$HDF5_TEST" "$DATA" "$@" -o "$prefix"
dials.import "${prefix}_master.h5" | tee import.log
grep -q "num images: 100" import.log || { echo "DIALS did not read all 100 images"; return 1; }
xia2.ssx image="${prefix}_master.h5" $CELL
}
dials_legacy() { dials_case test01 legacy -n100; }
dials_vds() { dials_case test02 vds -n100 -f25 -V; }
# -S writes <prefix>_master.h5, not <prefix>.h5 - naming this wrong meant xia2.ssx was
# pointed at a file that never existed, so the integrated format went untested.
dials_single() { dials_case test03 single -n100 -S; }
# Everything above writes an UNTILTED detector, where every plausible encoding of the tilt
# agrees exactly - so none of it constrains the NXmx transformation chain. This does.
dials_tilted() {
source /opt/dials-v3-27-0/dials_env.sh
rm -rf test04; mkdir -p test04; cd test04
"$HDF5_TEST" "$DATA" -n1 -S -o geom
dials.python "$GITHUB_WORKSPACE/tests/nxmx_geometry_dials_test.py" geom_master.h5
}
# -- XDS, once per read plugin -----------------------------------------------------
# Each case clears its directory first. The runner reuses the workspace volume, so a
# leftover .h5/.LP from an earlier run could let XDS "pass" on files this case never
# wrote -- which is what the first case of two of these jobs used to risk.
# Indexing must succeed, not just run: a first IDXREF refinement above 2 px is a false
# minimum (Durin fell into one at 6 px while XDS.INP started 7 px off the beam centre). And
# SPOT.XDS is kept per layout: a plugin must find the same spots in every layout, since they
# all hold the same pixels. Only COLSPOT's columns (x, y, frame, intensity) are compared:
# IDXREF writes its h,k,l back into the file, and its choice of basis is not deterministic.
xds_case() { # <dir> <layout> <jfjoch_hdf5_test flags...>
local dir=$1 layout=$2; shift 2
cd "tests/$dir"
rm -f ./*.h5 ./*.LP ./*.HKL ./SPOT.XDS "./SPOT.XDS.$layout"
"$HDF5_TEST" "$DATA" "$@"
/opt/xds/xds_par
test -f CORRECT.LP
sd=$(grep -m1 'STANDARD DEVIATION OF SPOT' IDXREF.LP | awk '{print $NF}')
echo "IDXREF spot position SD: $sd px"
awk -v sd="$sd" 'BEGIN { exit !(sd < 2.0) }' || { echo "IDXREF converged to a false minimum"; return 1; }
cp SPOT.XDS "SPOT.XDS.$layout"
}
same_spots() { # <dir> <layouts...>: every layout gave the same SPOT.XDS
local dir=$1 first=$2; shift 2
for l in "$@"; do
[ "$(awk '{print $1, $2, $3, $4}' "tests/$dir/SPOT.XDS.$first" | md5sum)" = \
"$(awk '{print $1, $2, $3, $4}' "tests/$dir/SPOT.XDS.$l" | md5sum)" ] \
|| { echo "$dir: SPOT.XDS of $l differs from $first"; return 1; }
done
}
durin_legacy() { xds_case xds_durin legacy -n25 -f10; }
durin_vds() { xds_case xds_durin vds -n25 -f10 -V; }
durin_single() { xds_case xds_durin single -n25 -S; }
jfjoch_legacy() { xds_case xds legacy -n25 -f10; }
jfjoch_vds() { xds_case xds vds -n25 -f10 -V; }
jfjoch_single() { xds_case xds single -n25 -S; }
# Neggia cannot read a VDS master, so it gets the two layouts it supports.
neggia_legacy() { xds_case xds_neggia legacy -n25 -f10; }
neggia_single() { xds_case xds_neggia single -n25 -S; }
durin_layouts() { same_spots xds_durin legacy vds single; }
jfjoch_layouts() { same_spots xds legacy vds single; }
neggia_layouts() { same_spots xds_neggia legacy single; }
fail=0
summary=""
for t in dials_legacy dials_vds dials_single dials_tilted \
durin_legacy durin_vds durin_single \
jfjoch_legacy jfjoch_vds jfjoch_single \
neggia_legacy neggia_single \
durin_layouts jfjoch_layouts neggia_layouts; do
echo "::group::$t"
# Run the subshell as a command of its own and test $? afterwards. A subshell used
# directly as an `if` condition has its `set -e` ignored, so the case would carry on
# past its first failure and report only the last command's status.
( set -e; cd "$GITHUB_WORKSPACE"; "$t" )
rc=$?
if [ "$rc" -eq 0 ]; then
summary="$summary PASS $t"$'\n'
else
summary="$summary FAIL $t (exit $rc)"$'\n'; fail=1
fi
echo "::endgroup::"
done
echo "===== HDF5 consumer tests ====="
printf '%s' "$summary"
exit "$fail"
python-client:
name: Generate python client
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
steps:
- uses: actions/checkout@v4
- name: Build python client
shell: bash
run: bash gen_python_client.sh
- name: Upload Package to Gitea PyPI
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
run: |
twine upload -u __token__ -p ${{ secrets.CI_PYPI_TOKEN }} --skip-existing dist/*
twine upload -u __token__ -p ${{ secrets.PIP_REPOSITORY_API_TOKEN }} --repository-url https://gitea.psi.ch/api/packages/mx/pypi dist/*
documentation:
name: Build documentation
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
steps:
- uses: actions/checkout@v4
- name: Build documentation
shell: bash
run: bash make_doc.sh
- name: Upload documentation
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
run: |
git config --global user.email "filip.leonarski@psi.ch"
git config --global user.name "Filip Leonarski (Gitea)"
git checkout --orphan gitea-pages
git reset --hard
# Remove everything except .git and public
git clean -fdx -e public -e .git
# Move public contents to root and remove the now-empty public dir
shopt -s dotglob nullglob
mv public/* .
rm -rf public
git add .
git commit -m "Deploy site"
git push -f https://${{secrets.GITHUB_TOKEN}}@gitea.psi.ch/${{ github.repository }}.git gitea-pages
# ---- Release -------------------------------------------------------------------------------------
# Runs on every event, but only creates something when asked. Keeping it unconditional is what
# lets every other job depend on it with a plain `needs:` -- a skipped dependency would otherwise
# skip its dependents too. When it creates nothing it is just a checkout.
create-release:
name: Create release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
steps:
- uses: actions/checkout@v4
with:
persist-credentials: 'true' # Optional; should be the default
# The LFS objects are pulled only to attach the FPGA images below, so this stays off the
# critical path of an ordinary push -- where this job is a bare checkout and nothing else.
- name: Configure auth and fetch LFS
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
GITEA_TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
git lfs install --local
AUTH=$(git config --local http.${{ github.server_url }}/.extraheader)
git config --local --unset http.${{ github.server_url }}/.extraheader
git config --local http.${{ github.server_url }}/${{ github.repository }}.git/info/lfs/objects/batch.extraheader "$AUTH"
git lfs pull
- name: Create release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
shell: bash
run: |
python3 gitea_create_release.py