mx/MX_Pmodule
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

README: consolidate build tips, ship new-module skill in-repo

Browse Source 
Refresh to modbuild 2.1.3 (no build subcommand, real flag list, no
--dry-run/--check-mode), document the --tmpdir/--distdir /das recipe,
and collect the lessons from the ccp4 build: no errexit in hooks,
pipefail vs yes|, partial-download reuse, env contamination (module
purge + reload modbuild), slow shared-storage installs. The new-module
skill (SKILL.md + REFERENCE.md) now lives in .claude/skills/new-module
so Claude Code auto-discovers it and other agents can be pointed at the
same files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
duan_j
2026-08-18 21:42:47 +02:00
co-authored by Claude Fable 5
parent 62a0c97aa6
commit f21bc10dc0
3 changed files with 674 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+114 -27
View File
@@ -1,49 +1,136 @@
# MX
This project includes all MX modules in use.
All MX Pmodules in use at PSI. One top-level directory per module, holding
exactly 4 files:
All modules are build with modbuild/2.1.2
Documentation for how-to-build modules can be found in: https://github.com/Pmodules/Pmodules/wiki
```
<Module>/
build # #!/usr/bin/env modbuild — bash, overrides pbuild::* hooks
modulefile # #%Module1.0 — Tcl, runtime env
files/config.yaml # format:1 — group/relstage/versions/deps/urls
README.md # description + module-specific pitfalls
```
Modules build on the **Ra** cluster (RHEL8) into `/opt/psi/MX/<name>/<version>`.
Upstream docs: https://github.com/Pmodules/Pmodules/wiki
## Building on Ra
```bash
module use unstable && module load modbuild/2.1.3
cd <Module>
modbuild build <version>
# verify it loads:
module use MX unstable && module load <Module>/<version>
```
- There is **no `build` subcommand**: usage is `modbuild [BUILD_SCRIPT] [options]
<version>` — `build` above is the script's filename, so run inside the module dir.
- Flags: `--prep|--configure|--compile|--install|--all` (run *up to* that step),
`--clean-install` (remove existing module first), `-f`/`--force-rebuild`, `-j N`,
`--tmpdir=DIR`, `--distdir=DIR`, `-v`, `--debug`.
There is **no `--dry-run` and no `--check-mode`**.
- Big downloads/builds: the defaults `/var/tmp/$USER` (tmp) and
`~/.cache/Pmodules/distfiles` (downloads) are too small — put both on /das,
**adapting the p-group path to your own**:
```bash
modbuild build \
--tmpdir=/das/work/p21/p21515/.cache/Pmodules \
--distdir=/das/work/p21/p21515/.cache/Pmodules \
--clean-install -v --debug <version>
```
`curl: (23) Failure writing output to destination` during a download means
quota/disk full at the target dir, not a network error.
## Tips & tricks
1. Module names are `name/version`, never `name/name-version`.
2. New versions start `relstage: unstable`; promote to `stable` only after the
module builds AND loads. A stable module may only depend on stable modules.
3. **Contaminated environment**: previously loaded modules can leak env vars
into the build and make it fail in odd ways. Fix:
`module purge && module load modbuild/2.1.3`, then build again.
4. **"done" does not mean success**: modbuild has no errexit in build hooks —
failed commands scroll by and the build still finishes. Guard critical steps
with `|| std::die 42 "message"` and end `install` by checking the file your
modulefile needs actually exists.
5. Hooks run with `pipefail` + `nounset`: `yes | installer` dies via SIGPIPE —
use `printf 'y\ny\n' | installer`.
6. prep reuses any file already in the distfiles dir with **no integrity
check** — an interrupted download leaves a partial tarball that later fails
with `gzip: unexpected end of file`. Either `rm` it and re-run, pin
`shasums:`, or use a custom prep with `wget --continue` + `gzip -t`
(see `ccp4/build`).
7. Unbound-variable on load (cctbx/phenix-style env scripts): add
`puts stdout "set +x nounset"` before the `source` line in the modulefile.
8. Do not use `pbuild::add_to_group`, `use_cmake`, `use_autotools`,
`set_download_url` — removed, they `std::die`. Use `group:`, `urls:`,
`configure_with:` in config.yaml instead.
9. `/opt/psi` is shared storage: untarring/installing GBs runs 10–30+ min with
no output. Watch `du -sh /opt/psi/MX/<name>/<version>` from a second shell
instead of killing a healthy build.
10. Check the per-module README before building — module-specific pitfalls are
documented there (`ccp4/README.md` is the most battle-tested example).
## Building with an AI agent
The repo ships the **new-module** skill at `.claude/skills/new-module/`
(`SKILL.md` = workflow + decision tree, `REFERENCE.md` = full schema, hooks and
copy-paste templates). It scaffolds the 4 files for a new module from the right
install-type archetype and prints the Ra build recipe.
- **Claude Code**: opens the repo → skill is auto-discovered. Type
`/new-module <tool, version, download URL, special requirements>` or simply
describe the task ("package X as a module") and it triggers.
- **Other agents** (Cursor, Codex, generic LLM chat): the skill is plain
Markdown — point the agent at `.claude/skills/new-module/SKILL.md` first,
then `REFERENCE.md`, and ask it to follow them.
- The skill copies from real modules in this repo as ground truth (`xds`,
`careless`, `DIALS`, `ccp4`, ...) — keep those exemplary, and fold new
lessons back into the skill files so the next build starts smarter.
## Pmodules background (from the wiki)
- Modules are organized in hierarchical groups; all modules in this repo belong to the `MX` group (`module use MX`).
- Filesystem layout: modulefile at `$PREFIX/GROUP/modulefiles/NAME/VERSION`, installation at `$PREFIX/GROUP/NAME/VERSION`, per-version config at `$PREFIX/GROUP/modulefiles/NAME/.config-VERSION` (YAML).
- Release stages: **unstable**, **stable**, **deprecated**. Newly built modules land in unstable; opt in with `module use unstable`.
- Pmodules automatically sets `$NAME_DIR`, `$NAME_HOME`, `$NAME_PREFIX`, `$NAME_VERSION` on load and prepends standard directories (`$PREFIX/bin` to `PATH`, `$PREFIX/lib` to `LD_LIBRARY_PATH`, ...) if they exist -- no explicit `setenv`/`prepend-path` needed in the modulefile.
- Modules are organized in hierarchical groups; everything here is in the `MX`
group (`module use MX`).
- Filesystem layout: modulefile at `$PREFIX/GROUP/modulefiles/NAME/VERSION`,
installation at `$PREFIX/GROUP/NAME/VERSION`, per-version config at
`$PREFIX/GROUP/modulefiles/NAME/.config-VERSION` (YAML).
- Release stages: **unstable**, **stable**, **deprecated**. New builds land in
unstable; opt in with `module use unstable`.
- Pmodules automatically sets `$NAME_DIR`, `$NAME_HOME`, `$NAME_PREFIX`,
`$NAME_VERSION` on load and prepends standard directories (`$PREFIX/bin` to
`PATH`, ...) if they exist — no explicit `setenv`/`prepend-path` needed for
those.
## Permissions
No need to switch to an -adm account any more: members of the `unx-mx_adm` group can update the modules in `/opt/psi/MX` directly. If you run into permission problems, please let the admins know.
Members of the `unx-mx_adm` group can update `/opt/psi/MX` directly — no -adm
account needed. Permission problems: tell the admins.
## Tutorial branches (build-*)
Each participant works on their own `build-<module>` branch (e.g. `build-dials`). All of them start from `main`.
Each participant works on their own `build-<module>` branch (e.g.
`build-dials`), all starting from `main`.
To propagate new material from `main` to the `build-*` branches:
While a branch is still identical to main (fast-forward):
While the branches are still identical to main (fast-forward, no force needed):
```
git push origin main:build-dials main:build-buster main:build-careless main:build-crystfel main:build-phenix main:build-pymol main:build-autoproc main:build-xtrapol8
for b in dials buster careless crystfel phenix pymol autoproc xtrapol8; do
git branch -f build-$b main
done
```bash
git push origin main:build-dials main:build-buster ...
for b in dials buster ...; do git branch -f build-$b main; done
```
Once the branches have diverged (people committed on them), merge main into each instead:
Once branches have diverged, merge main into each instead:
```
for b in dials buster careless crystfel phenix pymol autoproc xtrapol8; do
```bash
for b in dials buster ...; do
git checkout build-$b && git merge main && git push
done
git checkout main
```
If `git push origin main:build-xxx` fails with "non-fast-forward", the branch has its own commits: switch to the merge loop, do not force-push over other people's work.
## Module tips
1. modules is name/version (not name/name-version)
2. Solution to unbound variables: `/opt/psi/MX/phenix/phenix-1.20-4459/build/setpaths.sh: line 4: LIBTBX_BUILD_RELOCATION_HINT: unbound variable` is to add `puts stdout "set +x nounset"` before the source statement in the modulefile
3. wait until the tutorial on 18th August 2026
If `git push origin main:build-xxx` fails with "non-fast-forward", the branch
has its own commits: use the merge loop, never force-push over other people's
work.