README: consolidate build tips, ship new-module skill in-repo

Refresh to modbuild 2.1.3 (no build subcommand, real flag list, no --dry-run/--check-mode), document the --tmpdir/--distdir /das recipe, and collect the lessons from the ccp4 build: no errexit in hooks, pipefail vs yes|, partial-download reuse, env contamination (module purge + reload modbuild), slow shared-storage installs. The new-module skill (SKILL.md + REFERENCE.md) now lives in .claude/skills/new-module so Claude Code auto-discovers it and other agents can be pointed at the same files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 21:42:47 +02:00
parent 62a0c97aa6
commit f21bc10dc0
3 changed files with 674 additions and 27 deletions
@@ -0,0 +1,459 @@
+# new-module — reference
+
+Ground truth is the Pmodules engine (`../../Pmodules/Pmodules/modbuild.in`,
+`libpbuild.bash`) and the worked modules in this repo. Read the cited real module before
+copying a template.
+
+## Variables available inside `build`
+
+| Var | Meaning |
+|---|---|
+| `$P` | module name (e.g. `xds`) |
+| `$V` | full version as given (e.g. `1.21.1-5286`) |
+| `$V_PKG` | version without the `-release` part (e.g. `1.21.1`) |
+| `$V_MAJOR` `$V_MINOR` `$V_PATCHLVL` `$V_RELEASE` | version components |
+| `$PREFIX` | install target = `/opt/psi/MX/<name>/<version>` |
+| `$SRC_DIR` | unpacked sources dir |
+| `$BUILD_DIR` | out-of-tree build dir (`=$SRC_DIR` if `compile_in_sourcetree`) |
+| `$BUILDBLOCK_DIR` | **the module directory itself** — where `build`, `files/`, committed tarballs live |
+| `$JOBS` | parallel make jobs (auto, capped at 10) |
+| `$PMODULES_TMPDIR` `$PMODULES_DISTFILESDIR` | build scratch / download cache |
+
+## Tmp & download-cache configuration
+
+Resolution order (`pm::read_config`, called by modbuild at startup):
+exported `PMODULES_TMPDIR`/`PMODULES_DISTFILESDIR` → `~/.Pmodules/Pmodules.yaml`
+(`tmp_dir:`, `download_dir:`) → system `Pmodules.yaml` → built-ins
+`/var/tmp/${USER}` and `~/.cache/Pmodules/distfiles`.
+
+- The user yaml is the durable per-user fix (big tarballs blow the home quota):
+  ```yaml
+  tmp_dir: /das/work/<group>/<proj>/.cache/Pmodules
+  download_dir: /das/work/<group>/<proj>/.cache/Pmodules/distfiles
+  ```
+- **Cannot be set in the build file**: `BUILD_ROOT`/`SRC_DIR`/`BUILD_DIR` are derived
+  from `PMODULES_TMPDIR` *before* `source "${BUILD_SCRIPT}"` (`libpbuild.bash:1608-1616`).
+- One-off override per run: `--tmpdir=DIR` / `--distdir=DIR` CLI flags.
+- Check with `modbuild build --prep <version>` (no `--dry-run` exists) — sources must
+  land under the new location.
+
+## Default `prep` download mechanics (worth knowing before overriding)
+
+For each `urls:` entry, prep looks for `name:` in `$PMODULES_DISTFILESDIR`, then
+`$BUILDBLOCK_DIR`, then `$BUILDBLOCK_DIR/files` — only if absent does it
+`curl --location --fail` into the distfiles dir. So downloads are cached across
+rebuilds, and hand-pre-staging a file (under the exact `name:`) into any of those
+dirs skips the download entirely. `shasums:` is optional — missing hash only logs
+an info line. `unpacker: none` just `cp`s the file into `$SRC_DIR` (no unpack).
+
+modbuild runs with `set -o pipefail -o nounset`: `yes | installer` fails via
+SIGPIPE — use `printf 'y\ny\n' | installer`.
+
+## `pbuild::` hooks
+
+Phases run in order `prep -> configure -> compile -> install`, each as a chain
+`pre_ -> (main) -> post_`. Override any; unset ones use the library default:
+
+| Hook | Default behaviour |
+|---|---|
+| `pbuild::prep` | download `urls`, verify `shasums`, unpack, apply `patch_files` |
+| `pbuild::configure` | run `./configure --prefix=$PREFIX $configure_args` or `cmake` (per `configure_with`) |
+| `pbuild::compile` | `make -j$JOBS -e` |
+| `pbuild::install` | `make install` |
+| `pbuild::pre_*` / `pbuild::post_*` | no-op (`:`) |
+
+Stub an unused hook with `pbuild::configure() { :; }`. A successful phase writes a stamp
+file; rerun forces with `-f`/`--clean-install`.
+
+**Removed — never use (they `std::die`):** `pbuild::add_to_group`, `pbuild::use_cmake`,
+`pbuild::use_autotools`, `pbuild::set_download_url`, `pbuild::set_sha256sum`,
+`pbuild::compile_in_sourcetree`. Express these in `config.yaml` instead (`group:`,
+`configure_with:`, `urls:`, `shasums:`, `compile_in_sourcetree:`).
+
+## `config.yaml` schema (`format: 1`, top key = module name)
+
+```yaml
+---
+format: 1
+<module>:
+  defaults:            # module-wide defaults
+    group: MX          # always MX for this repo
+    overlay: base      # standard overlay
+    relstage: unstable # unstable | stable | deprecated | remove
+    urls:              # optional — for downloaded sources
+      - url: https://.../<tool>-${V_PKG}.tar.gz   # $P/$V_PKG/$V_MAJOR expand (envsubst)
+        name: <tool>-${V_PKG}.tar.gz              # optional saved filename
+        strip_dirs: 1                             # optional tar --strip-components
+        unpacker: tar                             # tar | 7z | none
+    build_requires:    # optional — loaded only during build
+      - gcc/12.3.0
+    runtime_deps:      # optional — auto-loaded when the module is loaded
+      - phenix/1.21.1-5286
+  shasums:             # optional — required when compiling downloaded sources
+    <tool>-<version>.tar.gz: <sha256>
+  versions:            # keys: comma lists + brace expansion, e.g. 3.15  or  2.{35..37}.2
+    <version>:
+      config:          # single variant  (use `variants:` list for multi-host/arch)
+        relstage: unstable
+```
+
+Key facts: `defaults` values are inherited by every version; per-version `config:`
+overrides them. `runtime_deps` is how a module auto-loads another (Xtrapol8 → phenix).
+`build_requires`/`runtime_deps` entries are `module/version`. Full key list (30+):
+`../../Pmodules/Pmodules/modbuild.in` `Yaml_default_config` (~line 468).
+
+### Multi-host / multi-arch: `variants:`
+
+Replace `config:` with a `variants:` list; modbuild picks the one whose `systems:`
+(host-FQDN or OS-release regex) and `target_cpus:` match the build host. `target_cpus`:
+`x86_64 | arm64 | aarch64`. `kernels`: `linux | darwin`. Real example is `Rosetta` (uses
+merlin7 globs). On Ra, use `rhel8` (OS release) or a `ra-.*` host glob:
+
+```yaml
+  versions:
+    <version>:
+      variants:
+        - overlay: base
+          systems: [rhel8]        # or  [ra-.*]   — Ra, RHEL8, x86_64
+          target_cpus: [x86_64]
+          relstage: unstable
+          build_requires: [gcc/12.3.0, Python/3.9.21]
+          runtime_deps:  [gcc/12.3.0, Python/3.9.21]
+```
+
+## `modulefile` (Tcl)
+
+Cookie `#%Module1.0` (alias `#%Pmodule`). Always the 5 metadata directives, then runtime
+env. `$PREFIX`, `$P`, `$V` are available.
+
+```tcl
+#%Module1.0
+
+module-whatis       "one-line description"
+module-url          "https://homepage"
+module-license      "license name/url"
+module-maintainer   "Name <name@psi.ch>"
+
+module-help         "
+Longer help text.
+"
+
+# then ONE of, depending on how the tool exposes binaries:
+setenv        <TOOL>   $PREFIX/bin                        ;# expose via env var (xds, jfjoch)
+# prepend-path PATH    $PREFIX                             ;# binaries at PREFIX root (olex2)
+# puts stdout "source $PREFIX/<subdir>/<tool>_env.sh;"     ;# vendor env script (DIALS, phenix)
+```
+
+If a sourced env script trips `nounset`, precede it with
+`puts stdout "set +x nounset"`. Version-specialise as `modulefile-X.Y.Z` if needed.
+
+---
+
+## Archetype templates
+
+### 1. Committed tarball, no compile — copy `../xds/`
+
+`build`:
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::install() {
+    # tarball is committed in this module dir; --strip-components=1 drops the top folder
+    mkdir -p "${PREFIX}/bin"
+    cd "${PREFIX}/bin"
+    tar -xf "${BUILDBLOCK_DIR}/<Tool>-linux_x86_64.tar.gz" --strip-components=1
+}
+```
+`files/config.yaml`: no `urls`/`shasums` (source is the committed blob):
+```yaml
+---
+format: 1
+<module>:
+  defaults: {group: MX, overlay: base, relstage: unstable}
+  versions:
+    <version>:
+      config: {relstage: unstable}
+```
+`modulefile`: metadata + `setenv <TOOL> $PREFIX/bin`.
+
+**Zip variant — copy `../olex2/`** (`build` derives os/arch, unzips, `mv`s into PREFIX):
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::configure() {
+    os=$(uname -s | tr '[:upper:]' '[:lower:]'); arch=$(uname -m)
+    case "$arch" in x86_64|amd64) arch="64" ;; aarch64|arm64) arch="arm64" ;; esac
+    ZIP="${SRC_DIR}/${P}-${os}${arch}.zip"
+    cp -n "${BUILDBLOCK_DIR}/${P}-${os}${arch}.zip" "$ZIP"
+    unzip -o -q "$ZIP" -d "${SRC_DIR}"; rm -f "$ZIP"
+}
+pbuild::install() { mv "${SRC_DIR}/${P}/"* "${PREFIX}/"; }
+```
+modulefile adds `prepend-path PATH $PREFIX` (binaries sit at PREFIX root).
+
+### 2. Downloaded source tarball, no compile — copy `../Xtrapol8/`
+
+`build`:
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::install() {
+    mkdir -p "$PREFIX/bin"; cd "$PREFIX/bin"
+    tar -xf "${BUILDBLOCK_DIR}/<Tool>-${V_PKG}.tar.gz" --strip-components=1
+}
+```
+`files/config.yaml` — `urls:` with `name:` so the file matches what `install` untars;
+`runtime_deps` if the tool needs another module at load time:
+```yaml
+---
+format: 1
+<module>:
+  defaults:
+    group: MX
+    overlay: base
+    relstage: unstable
+    urls:
+      - url: https://github.com/<org>/<repo>/archive/refs/tags/v${V_PKG}.tar.gz
+        name: <Tool>-${V_PKG}.tar.gz
+  versions:
+    <version>:
+      config:
+        relstage: unstable
+        runtime_deps: [phenix/1.21.1-5286]
+```
+
+### 3. Downloaded source + compile — copy `../Rosetta/`
+
+`build`:
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::configure() { :; }
+pbuild::compile() {
+    cd "$SRC_DIR/main/source" || exit 1
+    ./scons.py -j"${JOBS}" mode=release bin
+}
+pbuild::install() {
+    cp -r "$SRC_DIR/main/"* "$PREFIX/"
+    mkdir -p "$PREFIX/bin"
+    install -m 755 "$SRC_DIR/main/source/bin/"* "$PREFIX/bin/"
+}
+```
+`files/config.yaml` — `urls:` **and** `shasums:` (compiled build must verify the source):
+```yaml
+---
+format: 1
+<module>:
+  defaults:
+    group: MX
+    overlay: base
+    relstage: unstable
+    urls:
+      - url: https://.../<tool>_source_${V_PKG}_bundle.tar.bz2
+  shasums:
+    <tool>_source_<version>_bundle.tar.bz2: <sha256>   # sha256sum <file>
+  versions:
+    <version>:
+      config:
+        relstage: unstable
+        build_requires: [gcc/12.3.0, Python/3.9.21]
+        runtime_deps:  [gcc/12.3.0, Python/3.9.21]
+```
+For per-host/per-arch builds use the `variants:` block shown above.
+
+### 4. git clone + build — copy `../pymol/`
+
+`build`:
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::prep() {
+    git clone --depth 1 --branch "v${V_PKG}" https://github.com/<org>/<repo>.git
+}
+pbuild::configure() {
+    # example: vendor an extra header-only dep into the source tree
+    git clone -b <tag> https://github.com/<org>/<dep>.git "$SRC_DIR/<dep>/"
+    cp -r "$SRC_DIR/<dep>/<dep>/" "$SRC_DIR/<repo>/include/"
+}
+pbuild::compile() {
+    cd "$SRC_DIR/<repo>/"
+    python3 setup.py install --prefix "$PREFIX/"
+}
+```
+`files/config.yaml` — pin the compiler/python toolchain in `build_requires`:
+```yaml
+  versions:
+    <version>:
+      config:
+        relstage: unstable
+        build_requires: [psi-python39/2021.11, gcc/9.3.0, cuda/11.0.3]
+```
+
+### 5. Vendor installer — copy `../DIALS/` (download) or `../phenix/` (login-gated)
+
+Download-and-run (`../DIALS/`):
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::install() {
+    cd "${PREFIX}"
+    wget https://<vendor>/<tool>-linux-x86_64.tar.xz
+    tar -xJf <tool>-linux-x86_64.tar.xz
+    cd <tool>-installer && ./install --prefix="$PREFIX"
+    cd "${PREFIX}" && rm -r <tool>-installer <tool>-linux-x86_64.tar.xz
+    ln -s <real-dir> <stable-alias>          # optional stable symlink
+}
+```
+modulefile: `puts stdout "source $PREFIX/<subdir>/<tool>_env.sh;"`.
+
+Login-gated installer (`../phenix/`) — pre-stage the installer into this module dir by
+hand (wget can't authenticate), then copy it in:
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::pre_configure() {
+    cp -r "${BUILDBLOCK_DIR}/<tool>-installer-${V}-.../"* "${SRC_DIR}"
+}
+pbuild::install() { cd "${SRC_DIR}" && ./install --prefix="${PREFIX}"; }
+```
+modulefile (note the nounset guard before the cctbx env script):
+```tcl
+puts stdout "set +x nounset"
+puts stdout "source /opt/psi/MX/<module>/${V}/<tool>-${V}/<tool>_env.sh"
+```
+Large installers: set `tmp_dir:`/`download_dir:` in `~/.Pmodules/Pmodules.yaml` (see
+"Tmp & download-cache configuration" above); an exported `PMODULES_TMPDIR` also works
+but must live in the shell, never in the build file.
+
+### 5b. Vendor tarball, session-gated URL + interactive setup script — copy `../ccp4/`
+
+For suites whose download link embeds an expiring session id (`sid=`) and whose setup
+script is interactive. `urls:` still handles download/caching/pre-staging; `unpacker:
+none` avoids unpacking ~20 GB twice — the setup script patches paths relative to its
+final location, so untar once, directly into `$PREFIX`:
+
+`files/config.yaml` url entry:
+```yaml
+    urls:
+      - url: "https://www.ccp4.ac.uk/download/download_file.php?os=linux&pkg=...&sid=<expires>"
+        name: ccp4-${V}-shelx-arp-x86_64.tar.gz
+        unpacker: none
+```
+`build` — every step die-guarded because modbuild has no errexit in hooks (a
+failed tar otherwise still ends in "done" with a broken PREFIX):
+```bash
+pbuild::install() {
+    cd "${PREFIX}"
+    tar -xzf "${SRC_DIR}/ccp4-${V}-shelx-arp-x86_64.tar.gz" || \
+        std::die 42 "ccp4: untar failed — cached tarball likely truncated; rm it from distfiles and re-run"
+    # 9-series tarball unpacks to ccp4-9, NOT ccp4-9.0 as the vendor doc implies — glob it
+    local -a top=( ccp4-*/ )
+    [[ -x "${top[0]}BINARY.setup" ]] || std::die 42 "ccp4: no BINARY.setup in ${top[0]}"
+    # pre-agree the licence so BINARY.setup never prompts; printf (not `yes`)
+    # because modbuild sets pipefail
+    touch "${PREFIX}/.agree2ccp4v6" "${HOME}/.agree2ccp4v6"
+    printf 'y\ny\ny\n' | "./${top[0]}BINARY.setup" || std::die 42 "ccp4: BINARY.setup failed"
+    ln -sfn "${top[0]%/}" ccp4   # version-independent alias
+    [[ -r ccp4/bin/ccp4.setup-sh ]] || std::die 42 "ccp4: setup-sh missing — modulefile would break"
+}
+```
+modulefile sources through the alias so minor updates don't break it:
+`puts stdout "source $PREFIX/ccp4/bin/ccp4.setup-sh;"`. No `shasums:` — the gated
+link can't be hashed ahead of the first fetch; pin it afterwards from the cached
+tarball. Expired sid → `curl --fail` dies cleanly; fix by refreshing the link in
+config.yaml or pre-staging the tarball in distfiles under the exact `name:`.
+An *interrupted* download instead leaves a partial file that prep reuses silently
+(`gzip: unexpected end of file` at untar). For huge/flaky downloads, skip `urls:`
+entirely and write a custom `pbuild::prep` doing `wget --continue` + `gzip -t`
+(resume + integrity gate; the builtin curl has neither) — `../ccp4/build` is the
+current ground truth for this. curl exit 23 ("Failure writing output") during a
+download = quota/disk full at the distfiles dir, not a network error.
+
+### 6. Python package via uv — copy `../careless/` (**preferred route for MX Python tools**)
+
+uv fetches a standalone CPython into `$PREFIX/python` (so the venv's interpreter
+symlinks never point at a home cache) and pip-installs the tool into `$PREFIX/venv`.
+MX conventions baked into this route:
+
+- **Shared uv cache `/opt/psi/MX/.uv-cache`** — `~/.cache/uv` hits the home disk
+  quota; the shared dir also reuses wheels across MX module builds.
+- `UV_PYTHON_INSTALL_DIR=$PREFIX/python` keeps the venv self-contained; verify with
+  `readlink -f $PREFIX/venv/bin/python` (must resolve inside `$PREFIX`).
+- modbuild's shell strips `~/.local/bin` from PATH, hence the `$UV` fallback.
+
+`build`:
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::prep() {
+    :
+}
+
+pbuild::configure() {
+    # modbuild's shell strips ~/.local/bin from PATH; override with UV=/path/to/uv if needed
+    UV="${UV:-$HOME/.local/bin/uv}"
+    export UV_CACHE_DIR="/opt/psi/MX/.uv-cache"
+    export UV_PYTHON_INSTALL_DIR="$PREFIX/python"
+    "$UV" venv --python 3.12 "$PREFIX/venv"
+}
+
+pbuild::compile() {
+    :
+}
+
+pbuild::install() {
+    UV="${UV:-$HOME/.local/bin/uv}"
+    export UV_CACHE_DIR="/opt/psi/MX/.uv-cache"
+    "$UV" pip install --python "$PREFIX/venv/bin/python" "<tool>==${V_PKG}"
+}
+```
+`files/config.yaml`: plain, no `urls`/`shasums` (pip fetches from PyPI).
+`modulefile`:
+```tcl
+setenv          <TOOL>_ENV    $PREFIX/venv
+prepend-path    PATH          $PREFIX/venv/bin
+```
+
+### 7. conda — reference `../alphafold/` (**avoid on Ra**)
+
+repo.anaconda.com is unreachable from Ra (GnuTLS SSL error), so the miniconda
+download below fails there — use the uv route (#6) instead. Kept for reference:
+alphafold installs a private miniconda into `$PREFIX` and builds a versioned env from a
+committed `environment.yml`. It is the outlier: it uses the **legacy `files/variants`**
+format and calls the removed `pbuild::add_to_group` — **do not copy those two**. For a
+new conda module, use a normal `files/config.yaml` (`group: MX`) and keep the build
+pattern:
+```bash
+#!/usr/bin/env modbuild
+
+pbuild::configure() {
+    mkdir -p "$PREFIX/miniconda"
+    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O "$PREFIX/miniconda/miniconda.sh"
+    bash "$PREFIX/miniconda/miniconda.sh" -b -u -p "$PREFIX/miniconda/"
+    "$PREFIX/miniconda/condabin/conda" env create --name "<tool>_$V" -f "$BUILDBLOCK_DIR/environment.yml"
+}
+pbuild::install() { cp -r "$BUILDBLOCK_DIR/bin" "$PREFIX/"; }   # runner scripts, if any
+```
+modulefile: `setenv <TOOL>_ENV "$PREFIX/miniconda/envs/<tool>_$V"`.
+
+---
+
+## Build & verify on Ra
+
+```bash
+module use unstable && module load modbuild/2.1.3    # the build tool (Pmodules/2.0.0 recipe is outdated)
+cd <Module>
+modbuild build <version>          # 'build' is the script filename, not a subcommand
+module use MX unstable && module load <Module>/<version>   # confirm it loads
+```
+
+Flags (2.1.3, from `modbuild --help`): `--prep|--configure|--compile|--install|--all`
+(run up to that step; `--all` adds cleanup), `--clean-install` (remove existing module
+first), `-f|--force-rebuild`, `-j N`, `--tmpdir=DIR`, `--distdir=DIR`,
+`--config-file=FILE`, `--disable-cleanup[-build|-src]`, `-v`, `--debug`,
+`--update-modulefiles`, `--update-relstage`, `--skip-subpkgs`, `--system`,
+`--with=P/V`, `--variant=VARIANT`. No `--dry-run`, no `--check-mode`.
+
+`relstage` visibility: `stable` shows by default; `unstable`/`deprecated` need
+`module use unstable` / `module use deprecated`. Promote unstable → stable by editing
+`config.yaml` and rerunning `modbuild build --update-modulefiles <version>`.
@@ -0,0 +1,101 @@
+---
+name: new-module
+description: Scaffold a new MX Pmodules 2.x module (build + modulefile + files/config.yaml + README) from the right install-type template, and print the Ra build+verify commands. Use when adding a new crystallography/MX tool to the MX_Pmodule repo, packaging software as a Pmodule, or when the user mentions modbuild, config.yaml, modulefile, or "build a module".
+trigger: /new-module
+---
+
+# new-module
+
+Scaffold a new module in this repo (`MX_Pmodule`). A module is a top-level directory
+named exactly the module name, holding **4 files**:
+
+```
+<Module>/
+  build              # #!/usr/bin/env modbuild  — bash, overrides pbuild::* hooks
+  modulefile         # #%Module1.0  — Tcl, runtime env
+  files/config.yaml  # format:1 — group/overlay/relstage/versions/deps/urls/shasums
+  README.md          # short description
+```
+
+Modules build on the **Ra** cluster (RHEL8), not merlin. `$PREFIX` installs to
+`/opt/psi/MX/<name>/<version>`.
+
+## Workflow
+
+1. **Gather inputs**: module name, version, homepage, license, maintainer, and **how the
+   software ships** — pick one row of the table below. If unclear, ask.
+2. **Scaffold** `<Module>/build`, `<Module>/modulefile`, `<Module>/files/config.yaml`,
+   `<Module>/README.md` by copying the matching archetype from
+   [REFERENCE.md](REFERENCE.md) and filling in name/version/urls. Look at the cited real
+   module (`../<Example>/`) as the ground truth before writing.
+3. **Downloaded sources**: run `sha256sum <tarball>` and put it under `shasums:` in
+   config.yaml (keyed by the exact downloaded filename).
+4. **Print the Ra build recipe** (do NOT run it — needs the cluster) and stop.
+
+## Install-type decision tree
+
+| Ships as | Copy from | config.yaml | build hooks | modulefile env |
+|---|---|---|---|---|
+| Committed tarball/zip, no compile | `xds`, `olex2`, `jfjoch_viewer` | no `urls`/`shasums`; commit the blob in `<Module>/` | `install` untars `$BUILDBLOCK_DIR/*.tar.gz --strip-components=1`, or unzip+`mv` | `setenv X $PREFIX/bin` |
+| Downloaded source tarball | `Xtrapol8` | `urls:` (+ `name:` if build expects a filename) | `install` untars the fetched file | `runtime_deps` |
+| Downloaded source + compile | `Rosetta` | `urls:` + `shasums:` | `compile` (scons/make), `install` (cp + `install -m755`) | `runtime_deps` |
+| git clone + build | `pymol` | `build_requires` | `prep` clone, `configure` extra deps, `compile` setup.py | — |
+| Vendor installer | `DIALS`, `phenix` | — | `install` wget + `./install --prefix=$PREFIX`, or `pre_configure` pre-stage if login-gated | `puts stdout "source .../env.sh"` |
+| Vendor tarball, session-gated URL + setup script | `ccp4` | `urls:` with `name:` + `unpacker: none` (sid links expire — README documents re-fetch/pre-stage) | `install` untars into `$PREFIX`, pre-agrees license file, runs vendor setup non-interactively | `puts stdout "source .../setup-sh"` via version-independent symlink |
+| pip package (uv route — **preferred for Python tools**) | `careless` | no `urls`/`shasums` (pip fetches) | `configure` uv venv into `$PREFIX`, `install` `uv pip install <tool>==$V_PKG`; shared cache `/opt/psi/MX/.uv-cache` | `prepend-path PATH $PREFIX/venv/bin` + `setenv *_ENV` |
+| conda | `alphafold` | (legacy `files/variants`) | `configure` miniconda + `conda env create` — **broken on Ra** (repo.anaconda.com SSL-fails); use the uv route | `setenv *_ENV` |
+
+Default config.yaml has **no `systems:` globs** (like xds/DIALS) so it builds on any Ra
+host. Use `variants:`/`systems:` only for per-host/per-arch builds (Rosetta) — and on Ra
+swap merlin7 host globs for a Ra pattern (`rhel8` OS-release, or a `ra-.*` host glob).
+
+## Ra build recipe (print this, filled in)
+
+```bash
+# On a Ra login node (e.g. ra-l-005), inside the repo:
+module use unstable && module load modbuild/2.1.3   # build tool (the old Pmodules/2.0.0 recipe is outdated)
+cd <Module>
+vi files/config.yaml                                # ensure the new <version> is listed
+modbuild build <version>                            # installs to /opt/psi/MX/<name>/<version>
+# verify it loads:
+module use MX unstable && module load <Module>/<version>
+```
+
+There is no `build` subcommand: usage is `modbuild [BUILD_SCRIPT] [options] <version>`,
+so `modbuild build <version>` works because `build` is the script's filename — run it
+inside the module dir. Real flags (2.1.3): `--prep|--configure|--compile|--install|--all`
+(run *up to* that step; `--all` adds cleanup), `--clean-install` (remove existing module
+first), `-f`/`--force-rebuild`, `-j N`, `--tmpdir=DIR` / `--distdir=DIR` (one-off
+tmp/download-cache override), `--config-file=FILE`, `-v`, `--debug`.
+There is **no `--dry-run` and no `--check-mode`** — to sanity-check config/paths, run
+`modbuild build --prep <version>` and inspect where sources landed.
+
+## Guardrails (do not skip)
+
+- Names are **`name/version`**, never `name/name-version`.
+- New versions start `relstage: unstable`; promote to `stable` only after it builds and
+  loads. A `stable` module may only depend on `stable` modules (modbuild enforces this).
+- If the modulefile `source`s a cctbx/phenix-style env script and the build hits
+  `... : unbound variable`, add `puts stdout "set +x nounset"` **before** the source line.
+- Do **not** use `pbuild::add_to_group`, `use_cmake`, `use_autotools`, `set_download_url`
+  — they are removed in current Pmodules and will `std::die`. Set `group:` / `urls:` /
+  `configure_with:` in config.yaml instead. (alphafold's legacy call is not a model.)
+- Big builds/downloads (phenix, ccp4): tmp defaults to `/var/tmp/$USER`, download cache
+  to `~/.cache/Pmodules/distfiles` (home quota!). Durable fix is `~/.Pmodules/Pmodules.yaml`
+  with `tmp_dir:`/`download_dir:` — modbuild reads it on every run. An exported
+  `PMODULES_TMPDIR` or `--tmpdir=DIR`/`--distdir=DIR` overrides the yaml per run.
+  It can NOT be set inside the build file: `BUILD_ROOT`/`SRC_DIR` are computed before
+  the build script is sourced (`libpbuild.bash:1608-1616`). Verify with
+  `modbuild build --prep <version>` and check where the sources landed.
+- modbuild runs hooks with `pipefail` + `nounset`: `yes | cmd` fails (SIGPIPE) — use
+  `printf 'y\ny\n' | cmd`.
+- modbuild has **no errexit in hooks**: a failed command scrolls by and the build still
+  prints "done", installing the modulefile over a broken PREFIX. Guard every critical
+  hook step with `|| std::die 42 "<what and how to recover>"` and end `install` with an
+  existence check of the file the modulefile needs.
+- Interrupted downloads leave a **partial file in the distfiles dir which prep reuses
+  silently** (no integrity check unless `shasums:` is set). Symptom: `gzip: unexpected
+  end of file`. Fix: `rm` it there and re-run; verify big tarballs with `gzip -t` first;
+  add the `shasums:` entry once the first good download exists.
+
+Full schema, all `pbuild::` hooks, env vars, and copy-paste templates: [REFERENCE.md](REFERENCE.md).
@@ -1,49 +1,136 @@
 # MX

-This project includes all MX modules in use.
+All MX Pmodules in use at PSI. One top-level directory per module, holding
+exactly 4 files:

-All modules are build with modbuild/2.1.2
-Documentation for how-to-build modules can be found in: https://github.com/Pmodules/Pmodules/wiki
+```
+<Module>/
+  build              # #!/usr/bin/env modbuild — bash, overrides pbuild::* hooks
+  modulefile         # #%Module1.0 — Tcl, runtime env
+  files/config.yaml  # format:1 — group/relstage/versions/deps/urls
+  README.md          # description + module-specific pitfalls
+```
+
+Modules build on the **Ra** cluster (RHEL8) into `/opt/psi/MX/<name>/<version>`.
+Upstream docs: https://github.com/Pmodules/Pmodules/wiki
+
+## Building on Ra
+
+```bash
+module use unstable && module load modbuild/2.1.3
+cd <Module>
+modbuild build <version>
+# verify it loads:
+module use MX unstable && module load <Module>/<version>
+```
+
+- There is **no `build` subcommand**: usage is `modbuild [BUILD_SCRIPT] [options]
+  <version>` — `build` above is the script's filename, so run inside the module dir.
+- Flags: `--prep|--configure|--compile|--install|--all` (run *up to* that step),
+  `--clean-install` (remove existing module first), `-f`/`--force-rebuild`, `-j N`,
+  `--tmpdir=DIR`, `--distdir=DIR`, `-v`, `--debug`.
+  There is **no `--dry-run` and no `--check-mode`**.
+- Big downloads/builds: the defaults `/var/tmp/$USER` (tmp) and
+  `~/.cache/Pmodules/distfiles` (downloads) are too small — put both on /das,
+  **adapting the p-group path to your own**:
+
+  ```bash
+  modbuild build \
+      --tmpdir=/das/work/p21/p21515/.cache/Pmodules \
+      --distdir=/das/work/p21/p21515/.cache/Pmodules \
+      --clean-install -v --debug <version>
+  ```
+
+  `curl: (23) Failure writing output to destination` during a download means
+  quota/disk full at the target dir, not a network error.
+
+## Tips & tricks
+
+1. Module names are `name/version`, never `name/name-version`.
+2. New versions start `relstage: unstable`; promote to `stable` only after the
+   module builds AND loads. A stable module may only depend on stable modules.
+3. **Contaminated environment**: previously loaded modules can leak env vars
+   into the build and make it fail in odd ways. Fix:
+   `module purge && module load modbuild/2.1.3`, then build again.
+4. **"done" does not mean success**: modbuild has no errexit in build hooks —
+   failed commands scroll by and the build still finishes. Guard critical steps
+   with `|| std::die 42 "message"` and end `install` by checking the file your
+   modulefile needs actually exists.
+5. Hooks run with `pipefail` + `nounset`: `yes | installer` dies via SIGPIPE —
+   use `printf 'y\ny\n' | installer`.
+6. prep reuses any file already in the distfiles dir with **no integrity
+   check** — an interrupted download leaves a partial tarball that later fails
+   with `gzip: unexpected end of file`. Either `rm` it and re-run, pin
+   `shasums:`, or use a custom prep with `wget --continue` + `gzip -t`
+   (see `ccp4/build`).
+7. Unbound-variable on load (cctbx/phenix-style env scripts): add
+   `puts stdout "set +x nounset"` before the `source` line in the modulefile.
+8. Do not use `pbuild::add_to_group`, `use_cmake`, `use_autotools`,
+   `set_download_url` — removed, they `std::die`. Use `group:`, `urls:`,
+   `configure_with:` in config.yaml instead.
+9. `/opt/psi` is shared storage: untarring/installing GBs runs 10–30+ min with
+   no output. Watch `du -sh /opt/psi/MX/<name>/<version>` from a second shell
+   instead of killing a healthy build.
+10. Check the per-module README before building — module-specific pitfalls are
+    documented there (`ccp4/README.md` is the most battle-tested example).
+
+## Building with an AI agent
+
+The repo ships the **new-module** skill at `.claude/skills/new-module/`
+(`SKILL.md` = workflow + decision tree, `REFERENCE.md` = full schema, hooks and
+copy-paste templates). It scaffolds the 4 files for a new module from the right
+install-type archetype and prints the Ra build recipe.
+
+- **Claude Code**: opens the repo → skill is auto-discovered. Type
+  `/new-module <tool, version, download URL, special requirements>` or simply
+  describe the task ("package X as a module") and it triggers.
+- **Other agents** (Cursor, Codex, generic LLM chat): the skill is plain
+  Markdown — point the agent at `.claude/skills/new-module/SKILL.md` first,
+  then `REFERENCE.md`, and ask it to follow them.
+- The skill copies from real modules in this repo as ground truth (`xds`,
+  `careless`, `DIALS`, `ccp4`, ...) — keep those exemplary, and fold new
+  lessons back into the skill files so the next build starts smarter.

 ## Pmodules background (from the wiki)

- Modules are organized in hierarchical groups; all modules in this repo belong to the `MX` group (`module use MX`).
- Filesystem layout: modulefile at `$PREFIX/GROUP/modulefiles/NAME/VERSION`, installation at `$PREFIX/GROUP/NAME/VERSION`, per-version config at `$PREFIX/GROUP/modulefiles/NAME/.config-VERSION` (YAML).
- Release stages: **unstable**, **stable**, **deprecated**. Newly built modules land in unstable; opt in with `module use unstable`.
- Pmodules automatically sets `$NAME_DIR`, `$NAME_HOME`, `$NAME_PREFIX`, `$NAME_VERSION` on load and prepends standard directories (`$PREFIX/bin` to `PATH`, `$PREFIX/lib` to `LD_LIBRARY_PATH`, ...) if they exist -- no explicit `setenv`/`prepend-path` needed in the modulefile.
+- Modules are organized in hierarchical groups; everything here is in the `MX`
+  group (`module use MX`).
+- Filesystem layout: modulefile at `$PREFIX/GROUP/modulefiles/NAME/VERSION`,
+  installation at `$PREFIX/GROUP/NAME/VERSION`, per-version config at
+  `$PREFIX/GROUP/modulefiles/NAME/.config-VERSION` (YAML).
+- Release stages: **unstable**, **stable**, **deprecated**. New builds land in
+  unstable; opt in with `module use unstable`.
+- Pmodules automatically sets `$NAME_DIR`, `$NAME_HOME`, `$NAME_PREFIX`,
+  `$NAME_VERSION` on load and prepends standard directories (`$PREFIX/bin` to
+  `PATH`, ...) if they exist — no explicit `setenv`/`prepend-path` needed for
+  those.

 ## Permissions

-No need to switch to an -adm account any more: members of the `unx-mx_adm` group can update the modules in `/opt/psi/MX` directly. If you run into permission problems, please let the admins know.
+Members of the `unx-mx_adm` group can update `/opt/psi/MX` directly — no -adm
+account needed. Permission problems: tell the admins.

 ## Tutorial branches (build-*)

-Each participant works on their own `build-<module>` branch (e.g. `build-dials`). All of them start from `main`.
+Each participant works on their own `build-<module>` branch (e.g.
+`build-dials`), all starting from `main`.

-To propagate new material from `main` to the `build-*` branches:
+While a branch is still identical to main (fast-forward):

-While the branches are still identical to main (fast-forward, no force needed):
-
-```
-git push origin main:build-dials main:build-buster main:build-careless main:build-crystfel main:build-phenix main:build-pymol main:build-autoproc main:build-xtrapol8
-
-for b in dials buster careless crystfel phenix pymol autoproc xtrapol8; do
-  git branch -f build-$b main
-done
+```bash
+git push origin main:build-dials main:build-buster ...
+for b in dials buster ...; do git branch -f build-$b main; done
 ```

-Once the branches have diverged (people committed on them), merge main into each instead:
+Once branches have diverged, merge main into each instead:

-```
-for b in dials buster careless crystfel phenix pymol autoproc xtrapol8; do
+```bash
+for b in dials buster ...; do
  git checkout build-$b && git merge main && git push
 done
 git checkout main
 ```

-If `git push origin main:build-xxx` fails with "non-fast-forward", the branch has its own commits: switch to the merge loop, do not force-push over other people's work.
-
-## Module tips
-1. modules is name/version (not name/name-version)
-2. Solution to unbound variables: `/opt/psi/MX/phenix/phenix-1.20-4459/build/setpaths.sh: line 4: LIBTBX_BUILD_RELOCATION_HINT: unbound variable` is to add `puts stdout "set +x nounset"` before the source statement in the modulefile
-3. wait until the tutorial on 18th August 2026
+If `git push origin main:build-xxx` fails with "non-fast-forward", the branch
+has its own commits: use the merge loop, never force-push over other people's
+work.