Sync new-module skill with current practice; README agent caveats
Skill (local + repo copy): --tmpdir/--distdir now part of the standard recipe (never write big files to /var/tmp — small and shared on the login node), env-contamination and slow-shared-storage guardrails added, ccp4 archetype 5b updated to the wget --continue + gzip -t prep that is actually in ccp4/build. Main README: agent section notes the skill can be buggy (use with caution, corrections welcome, reading along is a good way to learn) and that new software or a new download site needs the vendor install page given as context; /var/tmp warning made explicit here and in ccp4/README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
@@ -21,19 +21,21 @@ copying a template.
|
||||
|
||||
## Tmp & download-cache configuration
|
||||
|
||||
**Never write big files to `/var/tmp`** — it is small and shared with everyone
|
||||
on the login node.
|
||||
|
||||
Resolution order (`pm::read_config`, called by modbuild at startup):
|
||||
exported `PMODULES_TMPDIR`/`PMODULES_DISTFILESDIR` → `~/.Pmodules/Pmodules.yaml`
|
||||
(`tmp_dir:`, `download_dir:`) → system `Pmodules.yaml` → built-ins
|
||||
`/var/tmp/${USER}` and `~/.cache/Pmodules/distfiles`.
|
||||
|
||||
- The user yaml is the durable per-user fix (big tarballs blow the home quota):
|
||||
```yaml
|
||||
tmp_dir: /das/work/<group>/<proj>/.cache/Pmodules
|
||||
download_dir: /das/work/<group>/<proj>/.cache/Pmodules/distfiles
|
||||
```
|
||||
- **Maintained route: always pass `--tmpdir=DIR` / `--distdir=DIR` CLI flags**
|
||||
pointing at the p-group work folder (defaults blow /var/tmp resp. home quota).
|
||||
- The user yaml (`tmp_dir:`/`download_dir:` keys) is the documented per-user
|
||||
default, but `download_dir` did not reliably take effect in practice — prefer
|
||||
the flags.
|
||||
- **Cannot be set in the build file**: `BUILD_ROOT`/`SRC_DIR`/`BUILD_DIR` are derived
|
||||
from `PMODULES_TMPDIR` *before* `source "${BUILD_SCRIPT}"` (`libpbuild.bash:1608-1616`).
|
||||
- One-off override per run: `--tmpdir=DIR` / `--distdir=DIR` CLI flags.
|
||||
- Check with `modbuild build --prep <version>` (no `--dry-run` exists) — sources must
|
||||
land under the new location.
|
||||
|
||||
@@ -328,24 +330,29 @@ but must live in the shell, never in the build file.
|
||||
### 5b. Vendor tarball, session-gated URL + interactive setup script — copy `../ccp4/`
|
||||
|
||||
For suites whose download link embeds an expiring session id (`sid=`) and whose setup
|
||||
script is interactive. `urls:` still handles download/caching/pre-staging; `unpacker:
|
||||
none` avoids unpacking ~20 GB twice — the setup script patches paths relative to its
|
||||
final location, so untar once, directly into `$PREFIX`:
|
||||
script is interactive. No `urls:` in config.yaml — a custom `pbuild::prep` downloads
|
||||
with `wget --continue` (resumes a partial 3.7 GB pull; the builtin curl has no
|
||||
retry/resume) and gates on `gzip -t` so a truncated file can never reach `install`.
|
||||
The setup script patches paths relative to its final location, so untar once,
|
||||
directly into `$PREFIX`. Every step die-guarded because modbuild has no errexit in
|
||||
hooks (a failed tar otherwise still ends in "done" with a broken PREFIX):
|
||||
|
||||
`files/config.yaml` url entry:
|
||||
```yaml
|
||||
urls:
|
||||
- url: "https://www.ccp4.ac.uk/download/download_file.php?os=linux&pkg=...&sid=<expires>"
|
||||
name: ccp4-${V}-shelx-arp-x86_64.tar.gz
|
||||
unpacker: none
|
||||
```
|
||||
`build` — every step die-guarded because modbuild has no errexit in hooks (a
|
||||
failed tar otherwise still ends in "done" with a broken PREFIX):
|
||||
```bash
|
||||
pbuild::prep() {
|
||||
local -r tarball="${PMODULES_DISTFILESDIR}/ccp4-${V}-shelx-arp-x86_64.tar.gz"
|
||||
local -r url='https://www.ccp4.ac.uk/download/download_file.php?os=...&sid=<expires>'
|
||||
if ! gzip -t "${tarball}" 2>/dev/null; then
|
||||
mkdir -p "${PMODULES_DISTFILESDIR}"
|
||||
wget --continue --tries=20 --waitretry=30 --output-document="${tarball}" "${url}" || \
|
||||
std::die 42 "ccp4: download failed — quota/disk full, or expired sid (fetch a fresh link)"
|
||||
gzip -t "${tarball}" || std::die 42 "ccp4: tarball truncated — re-run to resume"
|
||||
fi
|
||||
}
|
||||
|
||||
pbuild::install() {
|
||||
cd "${PREFIX}"
|
||||
tar -xzf "${SRC_DIR}/ccp4-${V}-shelx-arp-x86_64.tar.gz" || \
|
||||
std::die 42 "ccp4: untar failed — cached tarball likely truncated; rm it from distfiles and re-run"
|
||||
tar -xzf "${PMODULES_DISTFILESDIR}/ccp4-${V}-shelx-arp-x86_64.tar.gz" || \
|
||||
std::die 42 "ccp4: untar failed — rm the tarball from distfiles and re-run"
|
||||
# 9-series tarball unpacks to ccp4-9, NOT ccp4-9.0 as the vendor doc implies — glob it
|
||||
local -a top=( ccp4-*/ )
|
||||
[[ -x "${top[0]}BINARY.setup" ]] || std::die 42 "ccp4: no BINARY.setup in ${top[0]}"
|
||||
@@ -360,14 +367,11 @@ pbuild::install() {
|
||||
modulefile sources through the alias so minor updates don't break it:
|
||||
`puts stdout "source $PREFIX/ccp4/bin/ccp4.setup-sh;"`. No `shasums:` — the gated
|
||||
link can't be hashed ahead of the first fetch; pin it afterwards from the cached
|
||||
tarball. Expired sid → `curl --fail` dies cleanly; fix by refreshing the link in
|
||||
config.yaml or pre-staging the tarball in distfiles under the exact `name:`.
|
||||
An *interrupted* download instead leaves a partial file that prep reuses silently
|
||||
(`gzip: unexpected end of file` at untar). For huge/flaky downloads, skip `urls:`
|
||||
entirely and write a custom `pbuild::prep` doing `wget --continue` + `gzip -t`
|
||||
(resume + integrity gate; the builtin curl has neither) — `../ccp4/build` is the
|
||||
current ground truth for this. curl exit 23 ("Failure writing output") during a
|
||||
download = quota/disk full at the distfiles dir, not a network error.
|
||||
tarball. Expired sid → refresh the URL, or pre-stage a hand-downloaded tarball in
|
||||
the distfiles dir under the exact name prep expects. curl/wget "Failure writing
|
||||
output" during a download = quota/disk full at the distfiles dir, not a network
|
||||
error. Vendor setup phases (path patching, py-compile) add 15-30+ min of silent
|
||||
runtime on shared storage — slow is normal. `../ccp4/` is the ground truth.
|
||||
|
||||
### 6. Python package via uv — copy `../careless/` (**preferred route for MX Python tools**)
|
||||
|
||||
@@ -443,7 +447,10 @@ modulefile: `setenv <TOOL>_ENV "$PREFIX/miniconda/envs/<tool>_$V"`.
|
||||
```bash
|
||||
module use unstable && module load modbuild/2.1.3 # the build tool (Pmodules/2.0.0 recipe is outdated)
|
||||
cd <Module>
|
||||
modbuild build <version> # 'build' is the script filename, not a subcommand
|
||||
modbuild build \
|
||||
--tmpdir=/das/work/<pgroup>/.cache/Pmodules \
|
||||
--distdir=/das/work/<pgroup>/.cache/Pmodules \
|
||||
<version> # 'build' is the script filename, not a subcommand
|
||||
module use MX unstable && module load <Module>/<version> # confirm it loads
|
||||
```
|
||||
|
||||
|
||||
@@ -52,11 +52,17 @@ swap merlin7 host globs for a Ra pattern (`rhel8` OS-release, or a `ra-.*` host
|
||||
## Ra build recipe (print this, filled in)
|
||||
|
||||
```bash
|
||||
# On a Ra login node (e.g. ra-l-005), inside the repo:
|
||||
# On a Ra login node (e.g. ra-l-005), inside the repo clone on /das:
|
||||
module use unstable && module load modbuild/2.1.3 # build tool (the old Pmodules/2.0.0 recipe is outdated)
|
||||
cd <Module>
|
||||
vi files/config.yaml # ensure the new <version> is listed
|
||||
modbuild build <version> # installs to /opt/psi/MX/<name>/<version>
|
||||
# ALWAYS pass --tmpdir/--distdir (defaults are /var/tmp/$USER — shared, do not
|
||||
# fill it — and ~/.cache/Pmodules/distfiles — home quota too small).
|
||||
# Adapt the p-group path (p21/p21515) to the user's own:
|
||||
modbuild build \
|
||||
--tmpdir=/das/work/p21/p21515/.cache/Pmodules \
|
||||
--distdir=/das/work/p21/p21515/.cache/Pmodules \
|
||||
<version> # installs to /opt/psi/MX/<name>/<version>
|
||||
# verify it loads:
|
||||
module use MX unstable && module load <Module>/<version>
|
||||
```
|
||||
@@ -80,13 +86,20 @@ There is **no `--dry-run` and no `--check-mode`** — to sanity-check config/pat
|
||||
- Do **not** use `pbuild::add_to_group`, `use_cmake`, `use_autotools`, `set_download_url`
|
||||
— they are removed in current Pmodules and will `std::die`. Set `group:` / `urls:` /
|
||||
`configure_with:` in config.yaml instead. (alphafold's legacy call is not a model.)
|
||||
- Big builds/downloads (phenix, ccp4): tmp defaults to `/var/tmp/$USER`, download cache
|
||||
to `~/.cache/Pmodules/distfiles` (home quota!). Durable fix is `~/.Pmodules/Pmodules.yaml`
|
||||
with `tmp_dir:`/`download_dir:` — modbuild reads it on every run. An exported
|
||||
`PMODULES_TMPDIR` or `--tmpdir=DIR`/`--distdir=DIR` overrides the yaml per run.
|
||||
It can NOT be set inside the build file: `BUILD_ROOT`/`SRC_DIR` are computed before
|
||||
the build script is sourced (`libpbuild.bash:1608-1616`). Verify with
|
||||
`modbuild build --prep <version>` and check where the sources landed.
|
||||
- **Never write big files to `/var/tmp`** — small and shared with everyone on the
|
||||
login node. Tmp/download dirs: ALWAYS pass `--tmpdir=DIR`/`--distdir=DIR` (see recipe). The
|
||||
`~/.Pmodules/Pmodules.yaml` route (`tmp_dir:`/`download_dir:`) exists but
|
||||
`download_dir` did not reliably take effect in practice — CLI flags are the
|
||||
maintained route. It can NOT be set inside the build file: `BUILD_ROOT`/`SRC_DIR`
|
||||
are computed before the build script is sourced (`libpbuild.bash:1608-1616`).
|
||||
Verify with `modbuild build --prep <version>` and check where the sources landed.
|
||||
Already polluted the defaults? `rm -rf /var/tmp/$USER/<Module>-<version>` and
|
||||
`rm -rf ~/.cache/Pmodules/distfiles`.
|
||||
- **Contaminated environment**: previously loaded modules leak env vars into builds
|
||||
and cause odd failures — `module purge && module load modbuild/2.1.3`, retry.
|
||||
- `/opt/psi` is shared storage: untarring/installing GBs runs 10–30+ min with no
|
||||
output (plus more for vendor setup/py-compile phases). Watch
|
||||
`du -sh /opt/psi/MX/<name>/<version>` from a second shell; don't kill a healthy build.
|
||||
- modbuild runs hooks with `pipefail` + `nounset`: `yes | cmd` fails (SIGPIPE) — use
|
||||
`printf 'y\ny\n' | cmd`.
|
||||
- modbuild has **no errexit in hooks**: a failed command scrolls by and the build still
|
||||
|
||||
@@ -41,10 +41,13 @@ New to git? Start here: https://git-scm.com/doc
|
||||
`<Module>` below is one of the top-level directories of the cloned repo
|
||||
(e.g. `/das/work/p21/p21515/MX_Pmodule/ccp4`):
|
||||
|
||||
**WARNING: never write big files to `/var/tmp` — it is small and shared with
|
||||
everyone on the login node. Please, please do not fill /var/tmp.**
|
||||
|
||||
Always pass `--tmpdir` and `--distdir` pointing to your p-group work folder —
|
||||
the defaults are `/var/tmp/$USER` (shared with everyone on the login node —
|
||||
**do NOT fill it**) and `~/.cache/Pmodules/distfiles` (home quota is too
|
||||
small). Adapt the p-group path (`p21/p21515`) to your own:
|
||||
the defaults are `/var/tmp/$USER` (see warning above) and
|
||||
`~/.cache/Pmodules/distfiles` (home quota is too small). Adapt the p-group
|
||||
path (`p21/p21515`) to your own:
|
||||
|
||||
```bash
|
||||
module use unstable && module load modbuild/2.1.3
|
||||
@@ -100,6 +103,13 @@ install-type archetype and prints the Ra build recipe.
|
||||
- **Other agents** (Cursor, Codex, generic LLM chat): the skill is plain
|
||||
Markdown — point the agent at `.claude/skills/new-module/SKILL.md` first,
|
||||
then `REFERENCE.md`, and ask it to follow them.
|
||||
- **New software (or a known tool that moved to a new download site)**: give
|
||||
the agent the software's installation/download page as part of the context —
|
||||
the skill knows the repo's patterns, not vendor-specific install procedures.
|
||||
- **The skill can be buggy** — use it with caution and double-check what it
|
||||
scaffolds before building; corrections are welcome (edit
|
||||
`.claude/skills/new-module/` and commit). Reading along while it works is
|
||||
also a good way to learn how these modules are put together.
|
||||
- The skill copies from real modules in this repo as ground truth (`xds`,
|
||||
`careless`, `DIALS`, `ccp4`, ...) — keep those exemplary, and fold new
|
||||
lessons back into the skill files so the next build starts smarter.
|
||||
|
||||
+3
-3
@@ -18,9 +18,9 @@ macromolecular crystallography. https://www.ccp4.ac.uk/
|
||||
## Build on Ra
|
||||
|
||||
Build tmp and download cache MUST be on /das, not the defaults
|
||||
(/var/tmp is too small, the tarball alone blows the $HOME quota — a
|
||||
download dying with "Failure writing output to destination" means
|
||||
exactly that). Pass both dirs explicitly on the command line so there
|
||||
(/var/tmp is small and **shared with everyone on the login node — do
|
||||
not fill it**; the tarball alone blows the $HOME quota — a download
|
||||
dying with "Failure writing output to destination" means exactly that). Pass both dirs explicitly on the command line so there
|
||||
is no doubt where things land — **adapt the p-group path
|
||||
(`p21/p21515`) to your own**:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user