From f21bc10dc0abd4eaa85d608448b998572bd94123 Mon Sep 17 00:00:00 2001 From: Dawn Date: Tue, 18 Aug 2026 21:42:47 +0200 Subject: [PATCH] README: consolidate build tips, ship new-module skill in-repo Refresh to modbuild 2.1.3 (no build subcommand, real flag list, no --dry-run/--check-mode), document the --tmpdir/--distdir /das recipe, and collect the lessons from the ccp4 build: no errexit in hooks, pipefail vs yes|, partial-download reuse, env contamination (module purge + reload modbuild), slow shared-storage installs. The new-module skill (SKILL.md + REFERENCE.md) now lives in .claude/skills/new-module so Claude Code auto-discovers it and other agents can be pointed at the same files. Co-Authored-By: Claude Fable 5 --- .claude/skills/new-module/REFERENCE.md | 459 +++++++++++++++++++++++++ .claude/skills/new-module/SKILL.md | 101 ++++++ README.md | 141 ++++++-- 3 files changed, 674 insertions(+), 27 deletions(-) create mode 100644 .claude/skills/new-module/REFERENCE.md create mode 100644 .claude/skills/new-module/SKILL.md diff --git a/.claude/skills/new-module/REFERENCE.md b/.claude/skills/new-module/REFERENCE.md new file mode 100644 index 0000000..b4dcad8 --- /dev/null +++ b/.claude/skills/new-module/REFERENCE.md @@ -0,0 +1,459 @@ +# new-module — reference + +Ground truth is the Pmodules engine (`../../Pmodules/Pmodules/modbuild.in`, +`libpbuild.bash`) and the worked modules in this repo. Read the cited real module before +copying a template. + +## Variables available inside `build` + +| Var | Meaning | +|---|---| +| `$P` | module name (e.g. `xds`) | +| `$V` | full version as given (e.g. `1.21.1-5286`) | +| `$V_PKG` | version without the `-release` part (e.g. `1.21.1`) | +| `$V_MAJOR` `$V_MINOR` `$V_PATCHLVL` `$V_RELEASE` | version components | +| `$PREFIX` | install target = `/opt/psi/MX//` | +| `$SRC_DIR` | unpacked sources dir | +| `$BUILD_DIR` | out-of-tree build dir (`=$SRC_DIR` if `compile_in_sourcetree`) | +| `$BUILDBLOCK_DIR` | **the module directory itself** — where `build`, `files/`, committed tarballs live | +| `$JOBS` | parallel make jobs (auto, capped at 10) | +| `$PMODULES_TMPDIR` `$PMODULES_DISTFILESDIR` | build scratch / download cache | + +## Tmp & download-cache configuration + +Resolution order (`pm::read_config`, called by modbuild at startup): +exported `PMODULES_TMPDIR`/`PMODULES_DISTFILESDIR` → `~/.Pmodules/Pmodules.yaml` +(`tmp_dir:`, `download_dir:`) → system `Pmodules.yaml` → built-ins +`/var/tmp/${USER}` and `~/.cache/Pmodules/distfiles`. + +- The user yaml is the durable per-user fix (big tarballs blow the home quota): + ```yaml + tmp_dir: /das/work///.cache/Pmodules + download_dir: /das/work///.cache/Pmodules/distfiles + ``` +- **Cannot be set in the build file**: `BUILD_ROOT`/`SRC_DIR`/`BUILD_DIR` are derived + from `PMODULES_TMPDIR` *before* `source "${BUILD_SCRIPT}"` (`libpbuild.bash:1608-1616`). +- One-off override per run: `--tmpdir=DIR` / `--distdir=DIR` CLI flags. +- Check with `modbuild build --prep ` (no `--dry-run` exists) — sources must + land under the new location. + +## Default `prep` download mechanics (worth knowing before overriding) + +For each `urls:` entry, prep looks for `name:` in `$PMODULES_DISTFILESDIR`, then +`$BUILDBLOCK_DIR`, then `$BUILDBLOCK_DIR/files` — only if absent does it +`curl --location --fail` into the distfiles dir. So downloads are cached across +rebuilds, and hand-pre-staging a file (under the exact `name:`) into any of those +dirs skips the download entirely. `shasums:` is optional — missing hash only logs +an info line. `unpacker: none` just `cp`s the file into `$SRC_DIR` (no unpack). + +modbuild runs with `set -o pipefail -o nounset`: `yes | installer` fails via +SIGPIPE — use `printf 'y\ny\n' | installer`. + +## `pbuild::` hooks + +Phases run in order `prep -> configure -> compile -> install`, each as a chain +`pre_ -> (main) -> post_`. Override any; unset ones use the library default: + +| Hook | Default behaviour | +|---|---| +| `pbuild::prep` | download `urls`, verify `shasums`, unpack, apply `patch_files` | +| `pbuild::configure` | run `./configure --prefix=$PREFIX $configure_args` or `cmake` (per `configure_with`) | +| `pbuild::compile` | `make -j$JOBS -e` | +| `pbuild::install` | `make install` | +| `pbuild::pre_*` / `pbuild::post_*` | no-op (`:`) | + +Stub an unused hook with `pbuild::configure() { :; }`. A successful phase writes a stamp +file; rerun forces with `-f`/`--clean-install`. + +**Removed — never use (they `std::die`):** `pbuild::add_to_group`, `pbuild::use_cmake`, +`pbuild::use_autotools`, `pbuild::set_download_url`, `pbuild::set_sha256sum`, +`pbuild::compile_in_sourcetree`. Express these in `config.yaml` instead (`group:`, +`configure_with:`, `urls:`, `shasums:`, `compile_in_sourcetree:`). + +## `config.yaml` schema (`format: 1`, top key = module name) + +```yaml +--- +format: 1 +: + defaults: # module-wide defaults + group: MX # always MX for this repo + overlay: base # standard overlay + relstage: unstable # unstable | stable | deprecated | remove + urls: # optional — for downloaded sources + - url: https://.../-${V_PKG}.tar.gz # $P/$V_PKG/$V_MAJOR expand (envsubst) + name: -${V_PKG}.tar.gz # optional saved filename + strip_dirs: 1 # optional tar --strip-components + unpacker: tar # tar | 7z | none + build_requires: # optional — loaded only during build + - gcc/12.3.0 + runtime_deps: # optional — auto-loaded when the module is loaded + - phenix/1.21.1-5286 + shasums: # optional — required when compiling downloaded sources + -.tar.gz: + versions: # keys: comma lists + brace expansion, e.g. 3.15 or 2.{35..37}.2 + : + config: # single variant (use `variants:` list for multi-host/arch) + relstage: unstable +``` + +Key facts: `defaults` values are inherited by every version; per-version `config:` +overrides them. `runtime_deps` is how a module auto-loads another (Xtrapol8 → phenix). +`build_requires`/`runtime_deps` entries are `module/version`. Full key list (30+): +`../../Pmodules/Pmodules/modbuild.in` `Yaml_default_config` (~line 468). + +### Multi-host / multi-arch: `variants:` + +Replace `config:` with a `variants:` list; modbuild picks the one whose `systems:` +(host-FQDN or OS-release regex) and `target_cpus:` match the build host. `target_cpus`: +`x86_64 | arm64 | aarch64`. `kernels`: `linux | darwin`. Real example is `Rosetta` (uses +merlin7 globs). On Ra, use `rhel8` (OS release) or a `ra-.*` host glob: + +```yaml + versions: + : + variants: + - overlay: base + systems: [rhel8] # or [ra-.*] — Ra, RHEL8, x86_64 + target_cpus: [x86_64] + relstage: unstable + build_requires: [gcc/12.3.0, Python/3.9.21] + runtime_deps: [gcc/12.3.0, Python/3.9.21] +``` + +## `modulefile` (Tcl) + +Cookie `#%Module1.0` (alias `#%Pmodule`). Always the 5 metadata directives, then runtime +env. `$PREFIX`, `$P`, `$V` are available. + +```tcl +#%Module1.0 + +module-whatis "one-line description" +module-url "https://homepage" +module-license "license name/url" +module-maintainer "Name " + +module-help " +Longer help text. +" + +# then ONE of, depending on how the tool exposes binaries: +setenv $PREFIX/bin ;# expose via env var (xds, jfjoch) +# prepend-path PATH $PREFIX ;# binaries at PREFIX root (olex2) +# puts stdout "source $PREFIX//_env.sh;" ;# vendor env script (DIALS, phenix) +``` + +If a sourced env script trips `nounset`, precede it with +`puts stdout "set +x nounset"`. Version-specialise as `modulefile-X.Y.Z` if needed. + +--- + +## Archetype templates + +### 1. Committed tarball, no compile — copy `../xds/` + +`build`: +```bash +#!/usr/bin/env modbuild + +pbuild::install() { + # tarball is committed in this module dir; --strip-components=1 drops the top folder + mkdir -p "${PREFIX}/bin" + cd "${PREFIX}/bin" + tar -xf "${BUILDBLOCK_DIR}/-linux_x86_64.tar.gz" --strip-components=1 +} +``` +`files/config.yaml`: no `urls`/`shasums` (source is the committed blob): +```yaml +--- +format: 1 +: + defaults: {group: MX, overlay: base, relstage: unstable} + versions: + : + config: {relstage: unstable} +``` +`modulefile`: metadata + `setenv $PREFIX/bin`. + +**Zip variant — copy `../olex2/`** (`build` derives os/arch, unzips, `mv`s into PREFIX): +```bash +#!/usr/bin/env modbuild + +pbuild::configure() { + os=$(uname -s | tr '[:upper:]' '[:lower:]'); arch=$(uname -m) + case "$arch" in x86_64|amd64) arch="64" ;; aarch64|arm64) arch="arm64" ;; esac + ZIP="${SRC_DIR}/${P}-${os}${arch}.zip" + cp -n "${BUILDBLOCK_DIR}/${P}-${os}${arch}.zip" "$ZIP" + unzip -o -q "$ZIP" -d "${SRC_DIR}"; rm -f "$ZIP" +} +pbuild::install() { mv "${SRC_DIR}/${P}/"* "${PREFIX}/"; } +``` +modulefile adds `prepend-path PATH $PREFIX` (binaries sit at PREFIX root). + +### 2. Downloaded source tarball, no compile — copy `../Xtrapol8/` + +`build`: +```bash +#!/usr/bin/env modbuild + +pbuild::install() { + mkdir -p "$PREFIX/bin"; cd "$PREFIX/bin" + tar -xf "${BUILDBLOCK_DIR}/-${V_PKG}.tar.gz" --strip-components=1 +} +``` +`files/config.yaml` — `urls:` with `name:` so the file matches what `install` untars; +`runtime_deps` if the tool needs another module at load time: +```yaml +--- +format: 1 +: + defaults: + group: MX + overlay: base + relstage: unstable + urls: + - url: https://github.com///archive/refs/tags/v${V_PKG}.tar.gz + name: -${V_PKG}.tar.gz + versions: + : + config: + relstage: unstable + runtime_deps: [phenix/1.21.1-5286] +``` + +### 3. Downloaded source + compile — copy `../Rosetta/` + +`build`: +```bash +#!/usr/bin/env modbuild + +pbuild::configure() { :; } +pbuild::compile() { + cd "$SRC_DIR/main/source" || exit 1 + ./scons.py -j"${JOBS}" mode=release bin +} +pbuild::install() { + cp -r "$SRC_DIR/main/"* "$PREFIX/" + mkdir -p "$PREFIX/bin" + install -m 755 "$SRC_DIR/main/source/bin/"* "$PREFIX/bin/" +} +``` +`files/config.yaml` — `urls:` **and** `shasums:` (compiled build must verify the source): +```yaml +--- +format: 1 +: + defaults: + group: MX + overlay: base + relstage: unstable + urls: + - url: https://.../_source_${V_PKG}_bundle.tar.bz2 + shasums: + _source__bundle.tar.bz2: # sha256sum + versions: + : + config: + relstage: unstable + build_requires: [gcc/12.3.0, Python/3.9.21] + runtime_deps: [gcc/12.3.0, Python/3.9.21] +``` +For per-host/per-arch builds use the `variants:` block shown above. + +### 4. git clone + build — copy `../pymol/` + +`build`: +```bash +#!/usr/bin/env modbuild + +pbuild::prep() { + git clone --depth 1 --branch "v${V_PKG}" https://github.com//.git +} +pbuild::configure() { + # example: vendor an extra header-only dep into the source tree + git clone -b https://github.com//.git "$SRC_DIR//" + cp -r "$SRC_DIR///" "$SRC_DIR//include/" +} +pbuild::compile() { + cd "$SRC_DIR//" + python3 setup.py install --prefix "$PREFIX/" +} +``` +`files/config.yaml` — pin the compiler/python toolchain in `build_requires`: +```yaml + versions: + : + config: + relstage: unstable + build_requires: [psi-python39/2021.11, gcc/9.3.0, cuda/11.0.3] +``` + +### 5. Vendor installer — copy `../DIALS/` (download) or `../phenix/` (login-gated) + +Download-and-run (`../DIALS/`): +```bash +#!/usr/bin/env modbuild + +pbuild::install() { + cd "${PREFIX}" + wget https:///-linux-x86_64.tar.xz + tar -xJf -linux-x86_64.tar.xz + cd -installer && ./install --prefix="$PREFIX" + cd "${PREFIX}" && rm -r -installer -linux-x86_64.tar.xz + ln -s # optional stable symlink +} +``` +modulefile: `puts stdout "source $PREFIX//_env.sh;"`. + +Login-gated installer (`../phenix/`) — pre-stage the installer into this module dir by +hand (wget can't authenticate), then copy it in: +```bash +#!/usr/bin/env modbuild + +pbuild::pre_configure() { + cp -r "${BUILDBLOCK_DIR}/-installer-${V}-.../"* "${SRC_DIR}" +} +pbuild::install() { cd "${SRC_DIR}" && ./install --prefix="${PREFIX}"; } +``` +modulefile (note the nounset guard before the cctbx env script): +```tcl +puts stdout "set +x nounset" +puts stdout "source /opt/psi/MX//${V}/-${V}/_env.sh" +``` +Large installers: set `tmp_dir:`/`download_dir:` in `~/.Pmodules/Pmodules.yaml` (see +"Tmp & download-cache configuration" above); an exported `PMODULES_TMPDIR` also works +but must live in the shell, never in the build file. + +### 5b. Vendor tarball, session-gated URL + interactive setup script — copy `../ccp4/` + +For suites whose download link embeds an expiring session id (`sid=`) and whose setup +script is interactive. `urls:` still handles download/caching/pre-staging; `unpacker: +none` avoids unpacking ~20 GB twice — the setup script patches paths relative to its +final location, so untar once, directly into `$PREFIX`: + +`files/config.yaml` url entry: +```yaml + urls: + - url: "https://www.ccp4.ac.uk/download/download_file.php?os=linux&pkg=...&sid=" + name: ccp4-${V}-shelx-arp-x86_64.tar.gz + unpacker: none +``` +`build` — every step die-guarded because modbuild has no errexit in hooks (a +failed tar otherwise still ends in "done" with a broken PREFIX): +```bash +pbuild::install() { + cd "${PREFIX}" + tar -xzf "${SRC_DIR}/ccp4-${V}-shelx-arp-x86_64.tar.gz" || \ + std::die 42 "ccp4: untar failed — cached tarball likely truncated; rm it from distfiles and re-run" + # 9-series tarball unpacks to ccp4-9, NOT ccp4-9.0 as the vendor doc implies — glob it + local -a top=( ccp4-*/ ) + [[ -x "${top[0]}BINARY.setup" ]] || std::die 42 "ccp4: no BINARY.setup in ${top[0]}" + # pre-agree the licence so BINARY.setup never prompts; printf (not `yes`) + # because modbuild sets pipefail + touch "${PREFIX}/.agree2ccp4v6" "${HOME}/.agree2ccp4v6" + printf 'y\ny\ny\n' | "./${top[0]}BINARY.setup" || std::die 42 "ccp4: BINARY.setup failed" + ln -sfn "${top[0]%/}" ccp4 # version-independent alias + [[ -r ccp4/bin/ccp4.setup-sh ]] || std::die 42 "ccp4: setup-sh missing — modulefile would break" +} +``` +modulefile sources through the alias so minor updates don't break it: +`puts stdout "source $PREFIX/ccp4/bin/ccp4.setup-sh;"`. No `shasums:` — the gated +link can't be hashed ahead of the first fetch; pin it afterwards from the cached +tarball. Expired sid → `curl --fail` dies cleanly; fix by refreshing the link in +config.yaml or pre-staging the tarball in distfiles under the exact `name:`. +An *interrupted* download instead leaves a partial file that prep reuses silently +(`gzip: unexpected end of file` at untar). For huge/flaky downloads, skip `urls:` +entirely and write a custom `pbuild::prep` doing `wget --continue` + `gzip -t` +(resume + integrity gate; the builtin curl has neither) — `../ccp4/build` is the +current ground truth for this. curl exit 23 ("Failure writing output") during a +download = quota/disk full at the distfiles dir, not a network error. + +### 6. Python package via uv — copy `../careless/` (**preferred route for MX Python tools**) + +uv fetches a standalone CPython into `$PREFIX/python` (so the venv's interpreter +symlinks never point at a home cache) and pip-installs the tool into `$PREFIX/venv`. +MX conventions baked into this route: + +- **Shared uv cache `/opt/psi/MX/.uv-cache`** — `~/.cache/uv` hits the home disk + quota; the shared dir also reuses wheels across MX module builds. +- `UV_PYTHON_INSTALL_DIR=$PREFIX/python` keeps the venv self-contained; verify with + `readlink -f $PREFIX/venv/bin/python` (must resolve inside `$PREFIX`). +- modbuild's shell strips `~/.local/bin` from PATH, hence the `$UV` fallback. + +`build`: +```bash +#!/usr/bin/env modbuild + +pbuild::prep() { + : +} + +pbuild::configure() { + # modbuild's shell strips ~/.local/bin from PATH; override with UV=/path/to/uv if needed + UV="${UV:-$HOME/.local/bin/uv}" + export UV_CACHE_DIR="/opt/psi/MX/.uv-cache" + export UV_PYTHON_INSTALL_DIR="$PREFIX/python" + "$UV" venv --python 3.12 "$PREFIX/venv" +} + +pbuild::compile() { + : +} + +pbuild::install() { + UV="${UV:-$HOME/.local/bin/uv}" + export UV_CACHE_DIR="/opt/psi/MX/.uv-cache" + "$UV" pip install --python "$PREFIX/venv/bin/python" "==${V_PKG}" +} +``` +`files/config.yaml`: plain, no `urls`/`shasums` (pip fetches from PyPI). +`modulefile`: +```tcl +setenv _ENV $PREFIX/venv +prepend-path PATH $PREFIX/venv/bin +``` + +### 7. conda — reference `../alphafold/` (**avoid on Ra**) + +repo.anaconda.com is unreachable from Ra (GnuTLS SSL error), so the miniconda +download below fails there — use the uv route (#6) instead. Kept for reference: +alphafold installs a private miniconda into `$PREFIX` and builds a versioned env from a +committed `environment.yml`. It is the outlier: it uses the **legacy `files/variants`** +format and calls the removed `pbuild::add_to_group` — **do not copy those two**. For a +new conda module, use a normal `files/config.yaml` (`group: MX`) and keep the build +pattern: +```bash +#!/usr/bin/env modbuild + +pbuild::configure() { + mkdir -p "$PREFIX/miniconda" + wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O "$PREFIX/miniconda/miniconda.sh" + bash "$PREFIX/miniconda/miniconda.sh" -b -u -p "$PREFIX/miniconda/" + "$PREFIX/miniconda/condabin/conda" env create --name "_$V" -f "$BUILDBLOCK_DIR/environment.yml" +} +pbuild::install() { cp -r "$BUILDBLOCK_DIR/bin" "$PREFIX/"; } # runner scripts, if any +``` +modulefile: `setenv _ENV "$PREFIX/miniconda/envs/_$V"`. + +--- + +## Build & verify on Ra + +```bash +module use unstable && module load modbuild/2.1.3 # the build tool (Pmodules/2.0.0 recipe is outdated) +cd +modbuild build # 'build' is the script filename, not a subcommand +module use MX unstable && module load / # confirm it loads +``` + +Flags (2.1.3, from `modbuild --help`): `--prep|--configure|--compile|--install|--all` +(run up to that step; `--all` adds cleanup), `--clean-install` (remove existing module +first), `-f|--force-rebuild`, `-j N`, `--tmpdir=DIR`, `--distdir=DIR`, +`--config-file=FILE`, `--disable-cleanup[-build|-src]`, `-v`, `--debug`, +`--update-modulefiles`, `--update-relstage`, `--skip-subpkgs`, `--system`, +`--with=P/V`, `--variant=VARIANT`. No `--dry-run`, no `--check-mode`. + +`relstage` visibility: `stable` shows by default; `unstable`/`deprecated` need +`module use unstable` / `module use deprecated`. Promote unstable → stable by editing +`config.yaml` and rerunning `modbuild build --update-modulefiles `. diff --git a/.claude/skills/new-module/SKILL.md b/.claude/skills/new-module/SKILL.md new file mode 100644 index 0000000..48d595a --- /dev/null +++ b/.claude/skills/new-module/SKILL.md @@ -0,0 +1,101 @@ +--- +name: new-module +description: Scaffold a new MX Pmodules 2.x module (build + modulefile + files/config.yaml + README) from the right install-type template, and print the Ra build+verify commands. Use when adding a new crystallography/MX tool to the MX_Pmodule repo, packaging software as a Pmodule, or when the user mentions modbuild, config.yaml, modulefile, or "build a module". +trigger: /new-module +--- + +# new-module + +Scaffold a new module in this repo (`MX_Pmodule`). A module is a top-level directory +named exactly the module name, holding **4 files**: + +``` +/ + build # #!/usr/bin/env modbuild — bash, overrides pbuild::* hooks + modulefile # #%Module1.0 — Tcl, runtime env + files/config.yaml # format:1 — group/overlay/relstage/versions/deps/urls/shasums + README.md # short description +``` + +Modules build on the **Ra** cluster (RHEL8), not merlin. `$PREFIX` installs to +`/opt/psi/MX//`. + +## Workflow + +1. **Gather inputs**: module name, version, homepage, license, maintainer, and **how the + software ships** — pick one row of the table below. If unclear, ask. +2. **Scaffold** `/build`, `/modulefile`, `/files/config.yaml`, + `/README.md` by copying the matching archetype from + [REFERENCE.md](REFERENCE.md) and filling in name/version/urls. Look at the cited real + module (`..//`) as the ground truth before writing. +3. **Downloaded sources**: run `sha256sum ` and put it under `shasums:` in + config.yaml (keyed by the exact downloaded filename). +4. **Print the Ra build recipe** (do NOT run it — needs the cluster) and stop. + +## Install-type decision tree + +| Ships as | Copy from | config.yaml | build hooks | modulefile env | +|---|---|---|---|---| +| Committed tarball/zip, no compile | `xds`, `olex2`, `jfjoch_viewer` | no `urls`/`shasums`; commit the blob in `/` | `install` untars `$BUILDBLOCK_DIR/*.tar.gz --strip-components=1`, or unzip+`mv` | `setenv X $PREFIX/bin` | +| Downloaded source tarball | `Xtrapol8` | `urls:` (+ `name:` if build expects a filename) | `install` untars the fetched file | `runtime_deps` | +| Downloaded source + compile | `Rosetta` | `urls:` + `shasums:` | `compile` (scons/make), `install` (cp + `install -m755`) | `runtime_deps` | +| git clone + build | `pymol` | `build_requires` | `prep` clone, `configure` extra deps, `compile` setup.py | — | +| Vendor installer | `DIALS`, `phenix` | — | `install` wget + `./install --prefix=$PREFIX`, or `pre_configure` pre-stage if login-gated | `puts stdout "source .../env.sh"` | +| Vendor tarball, session-gated URL + setup script | `ccp4` | `urls:` with `name:` + `unpacker: none` (sid links expire — README documents re-fetch/pre-stage) | `install` untars into `$PREFIX`, pre-agrees license file, runs vendor setup non-interactively | `puts stdout "source .../setup-sh"` via version-independent symlink | +| pip package (uv route — **preferred for Python tools**) | `careless` | no `urls`/`shasums` (pip fetches) | `configure` uv venv into `$PREFIX`, `install` `uv pip install ==$V_PKG`; shared cache `/opt/psi/MX/.uv-cache` | `prepend-path PATH $PREFIX/venv/bin` + `setenv *_ENV` | +| conda | `alphafold` | (legacy `files/variants`) | `configure` miniconda + `conda env create` — **broken on Ra** (repo.anaconda.com SSL-fails); use the uv route | `setenv *_ENV` | + +Default config.yaml has **no `systems:` globs** (like xds/DIALS) so it builds on any Ra +host. Use `variants:`/`systems:` only for per-host/per-arch builds (Rosetta) — and on Ra +swap merlin7 host globs for a Ra pattern (`rhel8` OS-release, or a `ra-.*` host glob). + +## Ra build recipe (print this, filled in) + +```bash +# On a Ra login node (e.g. ra-l-005), inside the repo: +module use unstable && module load modbuild/2.1.3 # build tool (the old Pmodules/2.0.0 recipe is outdated) +cd +vi files/config.yaml # ensure the new is listed +modbuild build # installs to /opt/psi/MX// +# verify it loads: +module use MX unstable && module load / +``` + +There is no `build` subcommand: usage is `modbuild [BUILD_SCRIPT] [options] `, +so `modbuild build ` works because `build` is the script's filename — run it +inside the module dir. Real flags (2.1.3): `--prep|--configure|--compile|--install|--all` +(run *up to* that step; `--all` adds cleanup), `--clean-install` (remove existing module +first), `-f`/`--force-rebuild`, `-j N`, `--tmpdir=DIR` / `--distdir=DIR` (one-off +tmp/download-cache override), `--config-file=FILE`, `-v`, `--debug`. +There is **no `--dry-run` and no `--check-mode`** — to sanity-check config/paths, run +`modbuild build --prep ` and inspect where sources landed. + +## Guardrails (do not skip) + +- Names are **`name/version`**, never `name/name-version`. +- New versions start `relstage: unstable`; promote to `stable` only after it builds and + loads. A `stable` module may only depend on `stable` modules (modbuild enforces this). +- If the modulefile `source`s a cctbx/phenix-style env script and the build hits + `... : unbound variable`, add `puts stdout "set +x nounset"` **before** the source line. +- Do **not** use `pbuild::add_to_group`, `use_cmake`, `use_autotools`, `set_download_url` + — they are removed in current Pmodules and will `std::die`. Set `group:` / `urls:` / + `configure_with:` in config.yaml instead. (alphafold's legacy call is not a model.) +- Big builds/downloads (phenix, ccp4): tmp defaults to `/var/tmp/$USER`, download cache + to `~/.cache/Pmodules/distfiles` (home quota!). Durable fix is `~/.Pmodules/Pmodules.yaml` + with `tmp_dir:`/`download_dir:` — modbuild reads it on every run. An exported + `PMODULES_TMPDIR` or `--tmpdir=DIR`/`--distdir=DIR` overrides the yaml per run. + It can NOT be set inside the build file: `BUILD_ROOT`/`SRC_DIR` are computed before + the build script is sourced (`libpbuild.bash:1608-1616`). Verify with + `modbuild build --prep ` and check where the sources landed. +- modbuild runs hooks with `pipefail` + `nounset`: `yes | cmd` fails (SIGPIPE) — use + `printf 'y\ny\n' | cmd`. +- modbuild has **no errexit in hooks**: a failed command scrolls by and the build still + prints "done", installing the modulefile over a broken PREFIX. Guard every critical + hook step with `|| std::die 42 ""` and end `install` with an + existence check of the file the modulefile needs. +- Interrupted downloads leave a **partial file in the distfiles dir which prep reuses + silently** (no integrity check unless `shasums:` is set). Symptom: `gzip: unexpected + end of file`. Fix: `rm` it there and re-run; verify big tarballs with `gzip -t` first; + add the `shasums:` entry once the first good download exists. + +Full schema, all `pbuild::` hooks, env vars, and copy-paste templates: [REFERENCE.md](REFERENCE.md). diff --git a/README.md b/README.md index 7ecd342..08e63d2 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,136 @@ # MX -This project includes all MX modules in use. +All MX Pmodules in use at PSI. One top-level directory per module, holding +exactly 4 files: -All modules are build with modbuild/2.1.2 -Documentation for how-to-build modules can be found in: https://github.com/Pmodules/Pmodules/wiki +``` +/ + build # #!/usr/bin/env modbuild — bash, overrides pbuild::* hooks + modulefile # #%Module1.0 — Tcl, runtime env + files/config.yaml # format:1 — group/relstage/versions/deps/urls + README.md # description + module-specific pitfalls +``` + +Modules build on the **Ra** cluster (RHEL8) into `/opt/psi/MX//`. +Upstream docs: https://github.com/Pmodules/Pmodules/wiki + +## Building on Ra + +```bash +module use unstable && module load modbuild/2.1.3 +cd +modbuild build +# verify it loads: +module use MX unstable && module load / +``` + +- There is **no `build` subcommand**: usage is `modbuild [BUILD_SCRIPT] [options] + ` — `build` above is the script's filename, so run inside the module dir. +- Flags: `--prep|--configure|--compile|--install|--all` (run *up to* that step), + `--clean-install` (remove existing module first), `-f`/`--force-rebuild`, `-j N`, + `--tmpdir=DIR`, `--distdir=DIR`, `-v`, `--debug`. + There is **no `--dry-run` and no `--check-mode`**. +- Big downloads/builds: the defaults `/var/tmp/$USER` (tmp) and + `~/.cache/Pmodules/distfiles` (downloads) are too small — put both on /das, + **adapting the p-group path to your own**: + + ```bash + modbuild build \ + --tmpdir=/das/work/p21/p21515/.cache/Pmodules \ + --distdir=/das/work/p21/p21515/.cache/Pmodules \ + --clean-install -v --debug + ``` + + `curl: (23) Failure writing output to destination` during a download means + quota/disk full at the target dir, not a network error. + +## Tips & tricks + +1. Module names are `name/version`, never `name/name-version`. +2. New versions start `relstage: unstable`; promote to `stable` only after the + module builds AND loads. A stable module may only depend on stable modules. +3. **Contaminated environment**: previously loaded modules can leak env vars + into the build and make it fail in odd ways. Fix: + `module purge && module load modbuild/2.1.3`, then build again. +4. **"done" does not mean success**: modbuild has no errexit in build hooks — + failed commands scroll by and the build still finishes. Guard critical steps + with `|| std::die 42 "message"` and end `install` by checking the file your + modulefile needs actually exists. +5. Hooks run with `pipefail` + `nounset`: `yes | installer` dies via SIGPIPE — + use `printf 'y\ny\n' | installer`. +6. prep reuses any file already in the distfiles dir with **no integrity + check** — an interrupted download leaves a partial tarball that later fails + with `gzip: unexpected end of file`. Either `rm` it and re-run, pin + `shasums:`, or use a custom prep with `wget --continue` + `gzip -t` + (see `ccp4/build`). +7. Unbound-variable on load (cctbx/phenix-style env scripts): add + `puts stdout "set +x nounset"` before the `source` line in the modulefile. +8. Do not use `pbuild::add_to_group`, `use_cmake`, `use_autotools`, + `set_download_url` — removed, they `std::die`. Use `group:`, `urls:`, + `configure_with:` in config.yaml instead. +9. `/opt/psi` is shared storage: untarring/installing GBs runs 10–30+ min with + no output. Watch `du -sh /opt/psi/MX//` from a second shell + instead of killing a healthy build. +10. Check the per-module README before building — module-specific pitfalls are + documented there (`ccp4/README.md` is the most battle-tested example). + +## Building with an AI agent + +The repo ships the **new-module** skill at `.claude/skills/new-module/` +(`SKILL.md` = workflow + decision tree, `REFERENCE.md` = full schema, hooks and +copy-paste templates). It scaffolds the 4 files for a new module from the right +install-type archetype and prints the Ra build recipe. + +- **Claude Code**: opens the repo → skill is auto-discovered. Type + `/new-module ` or simply + describe the task ("package X as a module") and it triggers. +- **Other agents** (Cursor, Codex, generic LLM chat): the skill is plain + Markdown — point the agent at `.claude/skills/new-module/SKILL.md` first, + then `REFERENCE.md`, and ask it to follow them. +- The skill copies from real modules in this repo as ground truth (`xds`, + `careless`, `DIALS`, `ccp4`, ...) — keep those exemplary, and fold new + lessons back into the skill files so the next build starts smarter. ## Pmodules background (from the wiki) -- Modules are organized in hierarchical groups; all modules in this repo belong to the `MX` group (`module use MX`). -- Filesystem layout: modulefile at `$PREFIX/GROUP/modulefiles/NAME/VERSION`, installation at `$PREFIX/GROUP/NAME/VERSION`, per-version config at `$PREFIX/GROUP/modulefiles/NAME/.config-VERSION` (YAML). -- Release stages: **unstable**, **stable**, **deprecated**. Newly built modules land in unstable; opt in with `module use unstable`. -- Pmodules automatically sets `$NAME_DIR`, `$NAME_HOME`, `$NAME_PREFIX`, `$NAME_VERSION` on load and prepends standard directories (`$PREFIX/bin` to `PATH`, `$PREFIX/lib` to `LD_LIBRARY_PATH`, ...) if they exist -- no explicit `setenv`/`prepend-path` needed in the modulefile. +- Modules are organized in hierarchical groups; everything here is in the `MX` + group (`module use MX`). +- Filesystem layout: modulefile at `$PREFIX/GROUP/modulefiles/NAME/VERSION`, + installation at `$PREFIX/GROUP/NAME/VERSION`, per-version config at + `$PREFIX/GROUP/modulefiles/NAME/.config-VERSION` (YAML). +- Release stages: **unstable**, **stable**, **deprecated**. New builds land in + unstable; opt in with `module use unstable`. +- Pmodules automatically sets `$NAME_DIR`, `$NAME_HOME`, `$NAME_PREFIX`, + `$NAME_VERSION` on load and prepends standard directories (`$PREFIX/bin` to + `PATH`, ...) if they exist — no explicit `setenv`/`prepend-path` needed for + those. ## Permissions -No need to switch to an -adm account any more: members of the `unx-mx_adm` group can update the modules in `/opt/psi/MX` directly. If you run into permission problems, please let the admins know. +Members of the `unx-mx_adm` group can update `/opt/psi/MX` directly — no -adm +account needed. Permission problems: tell the admins. ## Tutorial branches (build-*) -Each participant works on their own `build-` branch (e.g. `build-dials`). All of them start from `main`. +Each participant works on their own `build-` branch (e.g. +`build-dials`), all starting from `main`. -To propagate new material from `main` to the `build-*` branches: +While a branch is still identical to main (fast-forward): -While the branches are still identical to main (fast-forward, no force needed): - -``` -git push origin main:build-dials main:build-buster main:build-careless main:build-crystfel main:build-phenix main:build-pymol main:build-autoproc main:build-xtrapol8 - -for b in dials buster careless crystfel phenix pymol autoproc xtrapol8; do - git branch -f build-$b main -done +```bash +git push origin main:build-dials main:build-buster ... +for b in dials buster ...; do git branch -f build-$b main; done ``` -Once the branches have diverged (people committed on them), merge main into each instead: +Once branches have diverged, merge main into each instead: -``` -for b in dials buster careless crystfel phenix pymol autoproc xtrapol8; do +```bash +for b in dials buster ...; do git checkout build-$b && git merge main && git push done git checkout main ``` -If `git push origin main:build-xxx` fails with "non-fast-forward", the branch has its own commits: switch to the merge loop, do not force-push over other people's work. - -## Module tips -1. modules is name/version (not name/name-version) -2. Solution to unbound variables: `/opt/psi/MX/phenix/phenix-1.20-4459/build/setpaths.sh: line 4: LIBTBX_BUILD_RELOCATION_HINT: unbound variable` is to add `puts stdout "set +x nounset"` before the source statement in the modulefile -3. wait until the tutorial on 18th August 2026 +If `git push origin main:build-xxx` fails with "non-fast-forward", the branch +has its own commits: use the merge loop, never force-push over other people's +work.