From 1c8a83aa948e3cdf432aa68bc46ca769c97295f3 Mon Sep 17 00:00:00 2001 From: Dawn Date: Tue, 18 Aug 2026 22:29:13 +0200 Subject: [PATCH] Sync new-module skill with current practice; README agent caveats MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Skill (local + repo copy): --tmpdir/--distdir now part of the standard recipe (never write big files to /var/tmp — small and shared on the login node), env-contamination and slow-shared-storage guardrails added, ccp4 archetype 5b updated to the wget --continue + gzip -t prep that is actually in ccp4/build. Main README: agent section notes the skill can be buggy (use with caution, corrections welcome, reading along is a good way to learn) and that new software or a new download site needs the vendor install page given as context; /var/tmp warning made explicit here and in ccp4/README. Co-Authored-By: Claude Fable 5 --- .claude/skills/new-module/REFERENCE.md | 65 ++++++++++++++------------ .claude/skills/new-module/SKILL.md | 31 ++++++++---- README.md | 16 +++++-- ccp4/README.md | 6 +-- 4 files changed, 74 insertions(+), 44 deletions(-) diff --git a/.claude/skills/new-module/REFERENCE.md b/.claude/skills/new-module/REFERENCE.md index b4dcad8..add5bc4 100644 --- a/.claude/skills/new-module/REFERENCE.md +++ b/.claude/skills/new-module/REFERENCE.md @@ -21,19 +21,21 @@ copying a template. ## Tmp & download-cache configuration +**Never write big files to `/var/tmp`** — it is small and shared with everyone +on the login node. + Resolution order (`pm::read_config`, called by modbuild at startup): exported `PMODULES_TMPDIR`/`PMODULES_DISTFILESDIR` → `~/.Pmodules/Pmodules.yaml` (`tmp_dir:`, `download_dir:`) → system `Pmodules.yaml` → built-ins `/var/tmp/${USER}` and `~/.cache/Pmodules/distfiles`. -- The user yaml is the durable per-user fix (big tarballs blow the home quota): - ```yaml - tmp_dir: /das/work///.cache/Pmodules - download_dir: /das/work///.cache/Pmodules/distfiles - ``` +- **Maintained route: always pass `--tmpdir=DIR` / `--distdir=DIR` CLI flags** + pointing at the p-group work folder (defaults blow /var/tmp resp. home quota). +- The user yaml (`tmp_dir:`/`download_dir:` keys) is the documented per-user + default, but `download_dir` did not reliably take effect in practice — prefer + the flags. - **Cannot be set in the build file**: `BUILD_ROOT`/`SRC_DIR`/`BUILD_DIR` are derived from `PMODULES_TMPDIR` *before* `source "${BUILD_SCRIPT}"` (`libpbuild.bash:1608-1616`). -- One-off override per run: `--tmpdir=DIR` / `--distdir=DIR` CLI flags. - Check with `modbuild build --prep ` (no `--dry-run` exists) — sources must land under the new location. @@ -328,24 +330,29 @@ but must live in the shell, never in the build file. ### 5b. Vendor tarball, session-gated URL + interactive setup script — copy `../ccp4/` For suites whose download link embeds an expiring session id (`sid=`) and whose setup -script is interactive. `urls:` still handles download/caching/pre-staging; `unpacker: -none` avoids unpacking ~20 GB twice — the setup script patches paths relative to its -final location, so untar once, directly into `$PREFIX`: +script is interactive. No `urls:` in config.yaml — a custom `pbuild::prep` downloads +with `wget --continue` (resumes a partial 3.7 GB pull; the builtin curl has no +retry/resume) and gates on `gzip -t` so a truncated file can never reach `install`. +The setup script patches paths relative to its final location, so untar once, +directly into `$PREFIX`. Every step die-guarded because modbuild has no errexit in +hooks (a failed tar otherwise still ends in "done" with a broken PREFIX): -`files/config.yaml` url entry: -```yaml - urls: - - url: "https://www.ccp4.ac.uk/download/download_file.php?os=linux&pkg=...&sid=" - name: ccp4-${V}-shelx-arp-x86_64.tar.gz - unpacker: none -``` -`build` — every step die-guarded because modbuild has no errexit in hooks (a -failed tar otherwise still ends in "done" with a broken PREFIX): ```bash +pbuild::prep() { + local -r tarball="${PMODULES_DISTFILESDIR}/ccp4-${V}-shelx-arp-x86_64.tar.gz" + local -r url='https://www.ccp4.ac.uk/download/download_file.php?os=...&sid=' + if ! gzip -t "${tarball}" 2>/dev/null; then + mkdir -p "${PMODULES_DISTFILESDIR}" + wget --continue --tries=20 --waitretry=30 --output-document="${tarball}" "${url}" || \ + std::die 42 "ccp4: download failed — quota/disk full, or expired sid (fetch a fresh link)" + gzip -t "${tarball}" || std::die 42 "ccp4: tarball truncated — re-run to resume" + fi +} + pbuild::install() { cd "${PREFIX}" - tar -xzf "${SRC_DIR}/ccp4-${V}-shelx-arp-x86_64.tar.gz" || \ - std::die 42 "ccp4: untar failed — cached tarball likely truncated; rm it from distfiles and re-run" + tar -xzf "${PMODULES_DISTFILESDIR}/ccp4-${V}-shelx-arp-x86_64.tar.gz" || \ + std::die 42 "ccp4: untar failed — rm the tarball from distfiles and re-run" # 9-series tarball unpacks to ccp4-9, NOT ccp4-9.0 as the vendor doc implies — glob it local -a top=( ccp4-*/ ) [[ -x "${top[0]}BINARY.setup" ]] || std::die 42 "ccp4: no BINARY.setup in ${top[0]}" @@ -360,14 +367,11 @@ pbuild::install() { modulefile sources through the alias so minor updates don't break it: `puts stdout "source $PREFIX/ccp4/bin/ccp4.setup-sh;"`. No `shasums:` — the gated link can't be hashed ahead of the first fetch; pin it afterwards from the cached -tarball. Expired sid → `curl --fail` dies cleanly; fix by refreshing the link in -config.yaml or pre-staging the tarball in distfiles under the exact `name:`. -An *interrupted* download instead leaves a partial file that prep reuses silently -(`gzip: unexpected end of file` at untar). For huge/flaky downloads, skip `urls:` -entirely and write a custom `pbuild::prep` doing `wget --continue` + `gzip -t` -(resume + integrity gate; the builtin curl has neither) — `../ccp4/build` is the -current ground truth for this. curl exit 23 ("Failure writing output") during a -download = quota/disk full at the distfiles dir, not a network error. +tarball. Expired sid → refresh the URL, or pre-stage a hand-downloaded tarball in +the distfiles dir under the exact name prep expects. curl/wget "Failure writing +output" during a download = quota/disk full at the distfiles dir, not a network +error. Vendor setup phases (path patching, py-compile) add 15-30+ min of silent +runtime on shared storage — slow is normal. `../ccp4/` is the ground truth. ### 6. Python package via uv — copy `../careless/` (**preferred route for MX Python tools**) @@ -443,7 +447,10 @@ modulefile: `setenv _ENV "$PREFIX/miniconda/envs/_$V"`. ```bash module use unstable && module load modbuild/2.1.3 # the build tool (Pmodules/2.0.0 recipe is outdated) cd -modbuild build # 'build' is the script filename, not a subcommand +modbuild build \ + --tmpdir=/das/work//.cache/Pmodules \ + --distdir=/das/work//.cache/Pmodules \ + # 'build' is the script filename, not a subcommand module use MX unstable && module load / # confirm it loads ``` diff --git a/.claude/skills/new-module/SKILL.md b/.claude/skills/new-module/SKILL.md index 48d595a..761df67 100644 --- a/.claude/skills/new-module/SKILL.md +++ b/.claude/skills/new-module/SKILL.md @@ -52,11 +52,17 @@ swap merlin7 host globs for a Ra pattern (`rhel8` OS-release, or a `ra-.*` host ## Ra build recipe (print this, filled in) ```bash -# On a Ra login node (e.g. ra-l-005), inside the repo: +# On a Ra login node (e.g. ra-l-005), inside the repo clone on /das: module use unstable && module load modbuild/2.1.3 # build tool (the old Pmodules/2.0.0 recipe is outdated) cd vi files/config.yaml # ensure the new is listed -modbuild build # installs to /opt/psi/MX// +# ALWAYS pass --tmpdir/--distdir (defaults are /var/tmp/$USER — shared, do not +# fill it — and ~/.cache/Pmodules/distfiles — home quota too small). +# Adapt the p-group path (p21/p21515) to the user's own: +modbuild build \ + --tmpdir=/das/work/p21/p21515/.cache/Pmodules \ + --distdir=/das/work/p21/p21515/.cache/Pmodules \ + # installs to /opt/psi/MX// # verify it loads: module use MX unstable && module load / ``` @@ -80,13 +86,20 @@ There is **no `--dry-run` and no `--check-mode`** — to sanity-check config/pat - Do **not** use `pbuild::add_to_group`, `use_cmake`, `use_autotools`, `set_download_url` — they are removed in current Pmodules and will `std::die`. Set `group:` / `urls:` / `configure_with:` in config.yaml instead. (alphafold's legacy call is not a model.) -- Big builds/downloads (phenix, ccp4): tmp defaults to `/var/tmp/$USER`, download cache - to `~/.cache/Pmodules/distfiles` (home quota!). Durable fix is `~/.Pmodules/Pmodules.yaml` - with `tmp_dir:`/`download_dir:` — modbuild reads it on every run. An exported - `PMODULES_TMPDIR` or `--tmpdir=DIR`/`--distdir=DIR` overrides the yaml per run. - It can NOT be set inside the build file: `BUILD_ROOT`/`SRC_DIR` are computed before - the build script is sourced (`libpbuild.bash:1608-1616`). Verify with - `modbuild build --prep ` and check where the sources landed. +- **Never write big files to `/var/tmp`** — small and shared with everyone on the + login node. Tmp/download dirs: ALWAYS pass `--tmpdir=DIR`/`--distdir=DIR` (see recipe). The + `~/.Pmodules/Pmodules.yaml` route (`tmp_dir:`/`download_dir:`) exists but + `download_dir` did not reliably take effect in practice — CLI flags are the + maintained route. It can NOT be set inside the build file: `BUILD_ROOT`/`SRC_DIR` + are computed before the build script is sourced (`libpbuild.bash:1608-1616`). + Verify with `modbuild build --prep ` and check where the sources landed. + Already polluted the defaults? `rm -rf /var/tmp/$USER/-` and + `rm -rf ~/.cache/Pmodules/distfiles`. +- **Contaminated environment**: previously loaded modules leak env vars into builds + and cause odd failures — `module purge && module load modbuild/2.1.3`, retry. +- `/opt/psi` is shared storage: untarring/installing GBs runs 10–30+ min with no + output (plus more for vendor setup/py-compile phases). Watch + `du -sh /opt/psi/MX//` from a second shell; don't kill a healthy build. - modbuild runs hooks with `pipefail` + `nounset`: `yes | cmd` fails (SIGPIPE) — use `printf 'y\ny\n' | cmd`. - modbuild has **no errexit in hooks**: a failed command scrolls by and the build still diff --git a/README.md b/README.md index ebc401d..3d91011 100644 --- a/README.md +++ b/README.md @@ -41,10 +41,13 @@ New to git? Start here: https://git-scm.com/doc `` below is one of the top-level directories of the cloned repo (e.g. `/das/work/p21/p21515/MX_Pmodule/ccp4`): +**WARNING: never write big files to `/var/tmp` — it is small and shared with +everyone on the login node. Please, please do not fill /var/tmp.** + Always pass `--tmpdir` and `--distdir` pointing to your p-group work folder — -the defaults are `/var/tmp/$USER` (shared with everyone on the login node — -**do NOT fill it**) and `~/.cache/Pmodules/distfiles` (home quota is too -small). Adapt the p-group path (`p21/p21515`) to your own: +the defaults are `/var/tmp/$USER` (see warning above) and +`~/.cache/Pmodules/distfiles` (home quota is too small). Adapt the p-group +path (`p21/p21515`) to your own: ```bash module use unstable && module load modbuild/2.1.3 @@ -100,6 +103,13 @@ install-type archetype and prints the Ra build recipe. - **Other agents** (Cursor, Codex, generic LLM chat): the skill is plain Markdown — point the agent at `.claude/skills/new-module/SKILL.md` first, then `REFERENCE.md`, and ask it to follow them. +- **New software (or a known tool that moved to a new download site)**: give + the agent the software's installation/download page as part of the context — + the skill knows the repo's patterns, not vendor-specific install procedures. +- **The skill can be buggy** — use it with caution and double-check what it + scaffolds before building; corrections are welcome (edit + `.claude/skills/new-module/` and commit). Reading along while it works is + also a good way to learn how these modules are put together. - The skill copies from real modules in this repo as ground truth (`xds`, `careless`, `DIALS`, `ccp4`, ...) — keep those exemplary, and fold new lessons back into the skill files so the next build starts smarter. diff --git a/ccp4/README.md b/ccp4/README.md index a110664..f78e644 100644 --- a/ccp4/README.md +++ b/ccp4/README.md @@ -18,9 +18,9 @@ macromolecular crystallography. https://www.ccp4.ac.uk/ ## Build on Ra Build tmp and download cache MUST be on /das, not the defaults -(/var/tmp is too small, the tarball alone blows the $HOME quota — a -download dying with "Failure writing output to destination" means -exactly that). Pass both dirs explicitly on the command line so there +(/var/tmp is small and **shared with everyone on the login node — do +not fill it**; the tarball alone blows the $HOME quota — a download +dying with "Failure writing output to destination" means exactly that). Pass both dirs explicitly on the command line so there is no doubt where things land — **adapt the p-group path (`p21/p21515`) to your own**: