From 9dea762f974ad033c97e4e475a0aee328ec91953 Mon Sep 17 00:00:00 2001 From: Dawn Date: Tue, 27 Oct 2026 11:09:58 +0100 Subject: [PATCH] crystfel: bundle HDF5 1.14.6, meson/ninja via uv lock (py3.12), add 0.13.0-rh9 Ra build node has no hdf5-devel/h5cc/hdf5.pc, so meson setup died with 'Dependency "hdf5" not found'. Build now compiles HDF5 (autotools, libdir lib64 to match crystfel's $ORIGIN/../lib64 RPATH) into $PREFIX and lets meson find it via $PREFIX/bin/h5cc. meson/ninja come from pyproject.toml + uv.lock synced into a scratch venv on uv-managed Python 3.12 (system python 3.9 is EOL for meson >= 1.12). Co-Authored-By: Claude Fable 5.1 --- crystfel/README.md | 55 +++++++++++++++++++++++------ crystfel/build | 72 +++++++++++++++++++++++++++++++------- crystfel/files/config.yaml | 12 +++++++ crystfel/pyproject.toml | 16 +++++++++ crystfel/uv.lock | 53 ++++++++++++++++++++++++++++ 5 files changed, 185 insertions(+), 23 deletions(-) create mode 100644 crystfel/pyproject.toml create mode 100644 crystfel/uv.lock diff --git a/crystfel/README.md b/crystfel/README.md index f8e5463..11ef699 100644 --- a/crystfel/README.md +++ b/crystfel/README.md @@ -16,15 +16,41 @@ sha256-verified). The upstream page https://desy.de/~twhite/crystfel/ hangs so it is not used. Built with meson per upstream INSTALL.md: `meson setup build --prefix=$PREFIX && meson compile -C build && meson install -C build`. -meson >= 0.60 is required (meson.build) and the Ra system meson is too old: -once per build user run `pip3 install --user meson ninja`; the build file calls -`~/.local/bin/meson` and dies with a hint if it is missing. +**meson/ninja via uv.** meson.build needs meson >= 0.60; the Ra system meson is +too old and the system python is 3.9 (EOL, meson >= 1.12 needs >= 3.10). +`pyproject.toml` + `uv.lock` pin meson and ninja for Python 3.12; the build runs +`uv sync --locked` into a throwaway venv in the build scratch dir, on a +uv-managed Python 3.12 (downloaded on first use, also into scratch). Nothing of +this lands in `$PREFIX`. `uv` must be on the build host (default +`~/.local/bin/uv`, `pip3 install --user uv`; override with `UV=/path/to/uv`). +Wheel cache is the shared `/opt/psi/MX/.uv-cache` (home quota). + +**HDF5 is bundled.** The Ra build node has no `hdf5-devel`, no `h5cc` and no +`hdf5.pc`, so `meson setup` died with `Dependency "hdf5" not found (tried +pkg-config, pkg-config and config-tool)`. The build therefore fetches the +HDF5 release tarball (GitHub, sha256-verified, `unpacker: none` so it is only +copied into `$SRC_DIR`), configures it with autotools into `$PREFIX` +(`--libdir=$PREFIX/lib64`, serial, no tests, zlib required, szip off) and puts +`$PREFIX/bin` on PATH so meson's hdf5 config-tool picks up `h5cc`. Autotools +on purpose: meson refuses a cmake-built `h5cc`. crystfel installs its own libs +to `lib64` too and bakes `$ORIGIN/../lib64` (binaries) / `$ORIGIN` (libcrystfel) +RPATHs, so no `LD_LIBRARY_PATH` is set by the modulefile; the install hook +runs `ldd indexamajig` and dies on any `not found`. `h5dump` & co. come along +on the module PATH. + +Build-host prerequisites (the build dies with a hint when one is missing): +gcc, make, flex, bison, zlib-devel, gsl-devel, git (meson wraps), uv. +Optional, only picked up if present at build time: gtk3/cairo/gdk-pixbuf +(GUI: `crystfel`, `cell_explorer`), fftw, zmq, msgpack, libtiff. Indexers +xgandalf/pinkindexer/fdip and millepede are meson subprojects fetched from +gitlab.desy.de during `meson setup` (optional; skipped if the fetch fails). Upstream warns: do NOT have any conda environment active while building. -Build-host prerequisites (meson, ninja, gcc, and the C libraries: hdf5, -gsl, and optionally cairo/gdk-pixbuf/gtk for the GUI, fftw, zmq) must be -available on the Ra build node -- add `build_requires:` entries in -`files/config.yaml` if matching Ra modules are needed. + +TODO: no HDF5 filter plugins (bitshuffle/LZ4, as written by jfjoch/Eiger) are +installed, so indexamajig cannot read such data until a plugin dir exists and +`HDF5_PLUGIN_PATH` is set in the modulefile. Needs a decision where the +plugins should live (own module vs. bundled here). Loading the module puts `$PREFIX/bin` on `PATH`. @@ -33,7 +59,8 @@ Loading the module puts `$PREFIX/bin` on `PATH`. ```sh module use unstable && module load modbuild/2.1.3 cd crystfel -modbuild build 0.13.0 +modbuild build --tmpdir=/das/work/p21/p21515/.cache/Pmodules \ + --distdir=/das/work/p21/p21515/.cache/Pmodules 0.13.0 module use MX unstable && module load crystfel/0.13.0 && indexamajig --version ``` @@ -43,7 +70,9 @@ module use MX unstable && module load crystfel/0.13.0 && indexamajig --version module use MX unstable && module load crystfel/0.13.0 which indexamajig indexamajig --version -cell_explorer # GUI check, needs X +ldd $(which indexamajig) | grep -E 'hdf5|crystfel' # both from $PREFIX/lib64 +h5cc -showconfig | grep -i deflate # zlib filter present +cell_explorer # GUI check, needs X and a gtk3 build ``` ## Adding a new version @@ -51,5 +80,9 @@ cell_explorer # GUI check, needs X Add the version under `versions:` and its tarball sha256 under `shasums:` in `files/config.yaml` (fetch the GitLab archive URL above with the new tag and `sha256sum` it; tag names on GitLab match the release version, no `v` -prefix), then on a Ra login node: -`modbuild build ` (modbuild/2.1.3). +prefix), then on a Ra login node: `modbuild build ` (modbuild/2.1.3). + +Bumping HDF5: change the GitHub url + `name:` and its `shasums:` entry in +`files/config.yaml` (stay on 1.14.x: 2.x drops autotools, see above). + +Bumping meson/ninja: edit `pyproject.toml`, run `uv lock`, commit `uv.lock`. diff --git a/crystfel/build b/crystfel/build index 45d989d..6765f56 100755 --- a/crystfel/build +++ b/crystfel/build @@ -1,27 +1,75 @@ #!/usr/bin/env modbuild # upstream INSTALL.md: meson setup build && meson compile && meson install. -# default pbuild::prep downloads + sha-verifies + unpacks the tarball (urls/shasums -# in files/config.yaml, strip_dirs:1 puts the sources at $SRC_DIR root). -# meson.build needs meson >= 0.60; Ra system meson is too old, so use a -# user-local one: `pip3 install --user meson ninja` (lands in ~/.local/bin). +# default pbuild::prep downloads + sha-verifies the tarballs in files/config.yaml: +# crystfel is unpacked at $SRC_DIR root (strip_dirs:1), the hdf5 tarball is only +# copied there (unpacker: none) and unpacked by hand into $SRC_DIR/hdf5 below. +# +# meson/ninja: Ra's system meson is too old (meson.build wants >= 0.60) and its +# system python is 3.9 (EOL; meson >= 1.12 needs >= 3.10) -> uv syncs the +# versions pinned in pyproject.toml/uv.lock into a throwaway venv on a +# uv-managed Python 3.12, both kept in the build scratch dir, not in $PREFIX. +# +# hdf5: the Ra build node has no hdf5-devel / h5cc / hdf5.pc, so meson's +# dependency('hdf5') fails. HDF5 is compiled into $PREFIX first (autotools, NOT +# cmake: meson's hdf5 config-tool rejects a cmake-built h5cc) and meson finds it +# through $PREFIX/bin/h5cc. --libdir=lib64 matches the $ORIGIN/../lib64 RPATH +# crystfel bakes into its binaries, so the module needs no LD_LIBRARY_PATH. +# # cd into SRC_DIR explicitly: hooks are not guaranteed to start there, and a # wrong cwd silently builds nothing (modbuild has no errexit in hooks). -MESON="${HOME}/.local/bin/meson" + +crystfel::env() { + # modbuild's shell strips ~/.local/bin from PATH; override with UV=/path/to/uv if needed + UV="${UV:-$HOME/.local/bin/uv}" + # ~/.cache/uv hits the home disk quota; shared cache reuses wheels across MX builds + export UV_CACHE_DIR="/opt/psi/MX/.uv-cache" + export UV_PROJECT_ENVIRONMENT="${SRC_DIR}/../meson-venv" + export UV_PYTHON_INSTALL_DIR="${SRC_DIR}/../uv-python" + # venv first (meson, ninja), then PREFIX (h5cc for meson's hdf5 lookup) + export PATH="${UV_PROJECT_ENVIRONMENT}/bin:${PREFIX}/bin:${PATH}" + cd "${SRC_DIR}" || std::die 42 "cannot cd to SRC_DIR=${SRC_DIR}" +} pbuild::configure() { - [[ -x "$MESON" ]] || std::die 42 "meson not found at $MESON - run: pip3 install --user meson ninja" - cd "${SRC_DIR}" || std::die 42 "cannot cd to SRC_DIR=${SRC_DIR}" - "$MESON" setup build --prefix="${PREFIX}" || std::die 42 "meson setup failed - check build/meson-logs/meson-log.txt" + crystfel::env + [[ -x "$UV" ]] || std::die 42 "uv not found at $UV - run: pip3 install --user uv (or set UV=/path/to/uv)" + for t in flex bison make; do + command -v "$t" >/dev/null || std::die 42 "$t not on PATH - crystfel/hdf5 need it (dnf install $t)" + done + # --locked: die if pyproject.toml and uv.lock disagree instead of silently re-resolving + "$UV" sync --locked --python 3.12 --project "${BUILDBLOCK_DIR}" \ + || std::die 42 "uv sync failed - no meson/ninja venv (PyPI/GitHub reachable? uv.lock stale -> uv lock)" + + # HDF5 into $PREFIX: serial, C only, no test suite (halves the build time). + # HDF5_CONFIG_ABORT makes configure die when zlib is missing instead of + # silently building without deflate; szip is not on Ra, so it is switched off + # rather than aborting on it. + mkdir -p hdf5 && tar -xzf hdf5-*.tar.gz -C hdf5 --strip-components=1 \ + || std::die 42 "hdf5 tarball unpack failed - is hdf5-*.tar.gz in ${SRC_DIR}? (prep copies it there)" + ( cd hdf5 \ + && HDF5_CONFIG_ABORT=yes ./configure --prefix="${PREFIX}" --libdir="${PREFIX}/lib64" \ + --enable-build-mode=production --disable-tests --with-zlib=yes --with-szlib=no \ + && make -j"${JOBS}" && make install ) \ + || std::die 42 "hdf5 build failed - see ${SRC_DIR}/hdf5/config.log (zlib-devel missing?)" + [[ -x "${PREFIX}/bin/h5cc" ]] || std::die 42 "h5cc missing from ${PREFIX}/bin - meson cannot find hdf5 without it" + + meson setup build --prefix="${PREFIX}" --libdir=lib64 \ + || std::die 42 "meson setup failed - check build/meson-logs/meson-log.txt" } pbuild::compile() { - cd "${SRC_DIR}" || std::die 42 "cannot cd to SRC_DIR=${SRC_DIR}" - "$MESON" compile -C build || std::die 42 "meson compile failed" + crystfel::env + meson compile -C build || std::die 42 "meson compile failed" } pbuild::install() { - cd "${SRC_DIR}" || std::die 42 "cannot cd to SRC_DIR=${SRC_DIR}" - "$MESON" install -C build || std::die 42 "meson install failed" + crystfel::env + meson install -C build || std::die 42 "meson install failed" [[ -x "${PREFIX}/bin/indexamajig" ]] || std::die 42 "indexamajig missing from ${PREFIX}/bin after install" + # ldd follows the whole closure (libcrystfel -> libhdf5): a "not found" here + # means the RPATH/libdir assumption above broke and the module would not run + if ldd "${PREFIX}/bin/indexamajig" | grep "not found"; then + std::die 42 "unresolved shared libs (above) - hdf5 libdir vs crystfel RPATH mismatch" + fi } diff --git a/crystfel/files/config.yaml b/crystfel/files/config.yaml index 0c0c446..523856f 100644 --- a/crystfel/files/config.yaml +++ b/crystfel/files/config.yaml @@ -8,12 +8,24 @@ crystfel: relstage: unstable # desy.de/~twhite release tarball URL times out (curl: (28)) from Ra and # elsewhere -> fetch the same release tag from DESY GitLab instead. + # hdf5: the Ra build node has no hdf5-devel/h5cc, so build compiles HDF5 + # into $PREFIX. unpacker: none -> prep only sha-verifies and copies the + # tarball into $SRC_DIR; it must not be unpacked here because strip_dirs + # applies to every tarball and both trees have a top-level src/ that would + # collide. Bump: change url+name here, shasum below (sha256sum the file). urls: - url: https://gitlab.desy.de/thomas.white/crystfel/-/archive/${V_PKG}/crystfel-${V_PKG}.tar.gz strip_dirs: 1 + - url: https://github.com/HDFGroup/hdf5/releases/download/hdf5_1.14.6/hdf5-1.14.6.tar.gz + name: hdf5-1.14.6.tar.gz + unpacker: none shasums: crystfel-0.13.0.tar.gz: 57c25352e099aee05c0e397e45c71f84b5c9dd1e5f595a118cce3a6f91ad62bf + hdf5-1.14.6.tar.gz: e4defbac30f50d64e1556374aa49e574417c9e72c6b1de7a4ff88c4b1bea6e9b versions: 0.13.0: config: relstage: unstable + 0.13.0-rh9: + config: + relstage: unstable diff --git a/crystfel/pyproject.toml b/crystfel/pyproject.toml new file mode 100644 index 0000000..a865a96 --- /dev/null +++ b/crystfel/pyproject.toml @@ -0,0 +1,16 @@ +# Build-time only: meson + ninja for ./build. Ra's system meson is too old for +# meson.build (>= 0.60) and its system python is 3.9 (EOL; meson >= 1.12 needs +# >= 3.10), so the build syncs these into a throwaway venv on a uv-managed +# Python 3.12. Nothing from here ends up in $PREFIX. +# After editing: `uv lock` and commit uv.lock (the build runs `uv sync --locked`). +[project] +name = "crystfel-build" +version = "0" +requires-python = "==3.12.*" +dependencies = [ + "meson>=1.3", # fs.relative_to() -> $ORIGIN/../lib64 RPATH in crystfel's meson.build + "ninja", +] + +[tool.uv] +package = false # no package in this dir, only the deps above diff --git a/crystfel/uv.lock b/crystfel/uv.lock new file mode 100644 index 0000000..05cd251 --- /dev/null +++ b/crystfel/uv.lock @@ -0,0 +1,53 @@ +version = 1 +revision = 3 +requires-python = "==3.12.*" + +[[package]] +name = "crystfel-build" +version = "0" +source = { virtual = "." } +dependencies = [ + { name = "meson" }, + { name = "ninja" }, +] + +[package.metadata] +requires-dist = [ + { name = "meson", specifier = ">=1.3" }, + { name = "ninja" }, +] + +[[package]] +name = "meson" +version = "1.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/c9/8c9983f4f3d9c4e22fd76bcf8cd053e4472aefae45fd044669b6daa34b53/meson-1.12.1.tar.gz", hash = "sha256:ab0a6ca09f8ef70c564c8241fb5a23957886a0b53fb58412b5e07eaf07dba743", size = 2524665, upload-time = "2026-09-22T22:58:28.875Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/57/99/36cd25f9598eef70db97577440a70a360af505f6f01c2338a18d1556079c/meson-1.12.1-py3-none-any.whl", hash = "sha256:930bc7542cbd9f57009e182fd014eba48cf1a0180a7b9006d2d32d8f168d8b02", size = 1108668, upload-time = "2026-09-22T22:58:26.047Z" }, +] + +[[package]] +name = "ninja" +version = "1.13.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ac/92/410b7917d16ab54c04b05cc32b9284803671d91cf79d33be6009c28d4ea8/ninja-1.13.2.tar.gz", hash = "sha256:525bfa3fc88aa30a4467df270fd5be6f9fcae8061d54d4df74ea1dc5abd5a975", size = 243739, upload-time = "2026-08-30T15:49:51.897Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b5/b8/90a9518f2264637084d6199bc20c2f3fb97fefc4c474d277720150c3bd6d/ninja-1.13.2-py3-none-macosx_10_9_universal2.whl", hash = "sha256:fd82e26c0706ad4ab88e5fdd26f3fab0a987a90f810160f6c322e752c6af298b", size = 306611, upload-time = "2026-08-30T15:49:28.267Z" }, + { url = "https://files.pythonhosted.org/packages/35/54/7368ce188625e39acc03ee362bb86cc9bfa6ad15e25c50889ae36e2889b3/ninja-1.13.2-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d775a5e43e9088f507a6250d57fcf5678eb31268c545feb5064ffeee33735622", size = 177180, upload-time = "2026-08-30T15:49:29.59Z" }, + { url = "https://files.pythonhosted.org/packages/80/1a/0b5601ece2a5de97253e7c7c442b70315333955593c2b55616fd17f1706f/ninja-1.13.2-py3-none-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:81d95081c0ad7c95f67bf682220361ed2a32659d7861b4766b03433c58f22516", size = 199485, upload-time = "2026-08-30T15:49:30.72Z" }, + { url = "https://files.pythonhosted.org/packages/48/23/fcbe234a66966e35928c47b86336f92a7612db4781665f4e5f5fddef9630/ninja-1.13.2-py3-none-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:227cbc3ae3e5e429692388103cae8c09451df086cd2d342dae0795af0d162547", size = 197676, upload-time = "2026-08-30T15:49:32.026Z" }, + { url = "https://files.pythonhosted.org/packages/24/eb/a6ca97ef0ff7bb8bdcb395ec65a716e65d7c1f40896c3afe0090bb3e1535/ninja-1.13.2-py3-none-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:1684c60d031c54c1d049541b64243c0c567dca5463dbd77682a8901780af293d", size = 187980, upload-time = "2026-08-30T15:49:33.372Z" }, + { url = "https://files.pythonhosted.org/packages/6e/53/ebfed7b689c338dd8ebeec9c0730c8d56821292f14e2536e5f3ef1a05744/ninja-1.13.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:65a24341b5ac09fcadcc37082660be40a94174e51a937fabf6e2cae26225fa2c", size = 183365, upload-time = "2026-08-30T15:49:34.53Z" }, + { url = "https://files.pythonhosted.org/packages/c7/d6/dcf06d7ab44ade992ae5aa1228feff317684b463a1bd47e8642b30ac922e/ninja-1.13.2-py3-none-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:aa3d2ae5706a2c4d1e93edc951d1c6cbb45107413c404f8fde1741239efbc9a0", size = 155089, upload-time = "2026-08-30T15:49:35.72Z" }, + { url = "https://files.pythonhosted.org/packages/e1/6b/6513c09c33382b17c05b4349b8e81437b18680d0d7ec6fb8f7edc29adda1/ninja-1.13.2-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:919572cbc3f233261ecd41fe1f3efc9d44aa02464a4588867e06a8b4f6f416ea", size = 152149, upload-time = "2026-08-30T15:49:36.971Z" }, + { url = "https://files.pythonhosted.org/packages/4d/70/d59fa4261f5ce586f43d8716de1da33d813afc116f0bf9173bf4cdbfdb2a/ninja-1.13.2-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:0e083700470c02ca154a855ae6d692d03564f5064cae52e113896f9ccc078418", size = 525392, upload-time = "2026-08-30T15:49:38.136Z" }, + { url = "https://files.pythonhosted.org/packages/37/04/c8c2dc5b2f5fee79a1691d490256b178b7e1af97d56769117422ae8a23cc/ninja-1.13.2-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:59d71c3e15b6b6f3d903eb0c27285544e0747ca59925ada7037bb1af781ad4b3", size = 466268, upload-time = "2026-08-30T15:49:39.479Z" }, + { url = "https://files.pythonhosted.org/packages/f3/6d/fce288647e53e96e0f0929a3c5b23986aae1b7101ea8889a3090237c5d13/ninja-1.13.2-py3-none-musllinux_1_2_i686.whl", hash = "sha256:f90f84affc441e219f15fe52532806c1c9dbd22fb66c3addddce88a3deaabab7", size = 605088, upload-time = "2026-08-30T15:49:40.826Z" }, + { url = "https://files.pythonhosted.org/packages/10/a2/d8eedd25d0ae80b9e874aea362013e67416877c8f732eb4d5e7c971fdb9c/ninja-1.13.2-py3-none-musllinux_1_2_ppc64le.whl", hash = "sha256:b2f687437fac460b27b7eadc99039b1163016fb4ba7276e2782a192d9f24ee0e", size = 610806, upload-time = "2026-08-30T15:49:42.168Z" }, + { url = "https://files.pythonhosted.org/packages/14/0f/696d96821fad1b5767fd311c1569dde8881a57412369bfe7b11bcbfde036/ninja-1.13.2-py3-none-musllinux_1_2_riscv64.whl", hash = "sha256:09de9ab04f7352f51570c73fd4913acb1e6c24be0a72cd8b80243d4d3ed04925", size = 533978, upload-time = "2026-08-30T15:49:43.451Z" }, + { url = "https://files.pythonhosted.org/packages/5d/69/28844ca579156776a202217a7cd66f60d06a0710a935e879bb89ce396ecc/ninja-1.13.2-py3-none-musllinux_1_2_s390x.whl", hash = "sha256:6a87bf42b123abe2f37737300185f0a303a891899da85d73a3613ee80547e578", size = 653822, upload-time = "2026-08-30T15:49:44.724Z" }, + { url = "https://files.pythonhosted.org/packages/f5/5f/c511f2952f94ab2966d60edd9c34e744ea32f2724b1184b62270bde55b3a/ninja-1.13.2-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:915bd482c4be41c75120fd67a22e0bb3f0fbb3bbc5f95b89787deadd59e27ef2", size = 544460, upload-time = "2026-08-30T15:49:46.29Z" }, + { url = "https://files.pythonhosted.org/packages/79/e7/fb0e828e89ac77ef77183a0834f17c4108e66088732fed86a3cc3c776a7a/ninja-1.13.2-py3-none-win32.whl", hash = "sha256:792cadbb9decfd1f776d4d0a6930feb46d08302eb57c176bcf26b09de5748e9f", size = 270319, upload-time = "2026-08-30T15:49:47.942Z" }, + { url = "https://files.pythonhosted.org/packages/3f/dd/3766b5f4d32e8a9b97d195496b0b01fbbe2e1a41669dab0cd6492a6ce199/ninja-1.13.2-py3-none-win_amd64.whl", hash = "sha256:1293f4078278b70d0ee4b6cc8f3a9e030656c9b2f59909970343c4fe76070118", size = 311798, upload-time = "2026-08-30T15:49:49.351Z" }, + { url = "https://files.pythonhosted.org/packages/b7/8d/59a31fa508070d042571d9d226b541a21000756817313f397da22287ad34/ninja-1.13.2-py3-none-win_arm64.whl", hash = "sha256:1db9852e528efa7702f5123969f86678663e46d57ff28ab13f5fd84d64a85fb1", size = 290620, upload-time = "2026-08-30T15:49:50.639Z" }, +]