173 lines
7.0 KiB
Markdown
173 lines
7.0 KiB
Markdown
# MX
|
|
|
|
All MX Pmodules in use at PSI. One top-level directory per module, holding
|
|
exactly 4 files:
|
|
|
|
```
|
|
<Module>/
|
|
build # #!/usr/bin/env modbuild — bash, overrides pbuild::* hooks
|
|
modulefile # #%Module1.0 — Tcl, runtime env
|
|
files/config.yaml # format:1 — group/relstage/versions/deps/urls
|
|
README.md # description + module-specific pitfalls
|
|
```
|
|
|
|
Modules build on the **Ra** cluster (RHEL8) into `/opt/psi/MX/<name>/<version>`.
|
|
Upstream docs: https://github.com/Pmodules/Pmodules/wiki
|
|
|
|
## Getting the repo
|
|
|
|
Clone into your work folder on Ra (once). SSH access to Gitea is recommended:
|
|
generate a key ([how-to](https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key))
|
|
and add the public key at https://gitea.psi.ch/user/settings/keys.
|
|
|
|
```bash
|
|
cd /das/work/p21/p21515/ # adapt the p-group path to your own
|
|
git clone git@gitea.psi.ch:mx/MX_Pmodule.git
|
|
cd /das/work/p21/p21515/MX_Pmodule
|
|
```
|
|
|
|
Work on your own branch, not on main:
|
|
|
|
```bash
|
|
git switch main && git pull # start from up-to-date main
|
|
git switch -c build-<module> # e.g. build-dials
|
|
git push -u origin build-<module>
|
|
```
|
|
|
|
New to git? Start here: https://git-scm.com/doc
|
|
|
|
## Building on Ra - DO NOT write into /var/tmp !
|
|
|
|
`<Module>` below is one of the top-level directories of the cloned repo
|
|
(e.g. `/das/work/p21/p21515/MX_Pmodule/ccp4`):
|
|
|
|
> **!!!WARNING!!!: never write big files to `/var/tmp` — it is small and shared with
|
|
everyone on the login node. Please, please do not fill /var/tmp.**
|
|
|
|
Always pass `--tmpdir` and `--distdir` pointing to your p-group work folder —
|
|
the defaults are `/var/tmp/$USER` (see warning above) and
|
|
`~/.cache/Pmodules/distfiles` (home quota is too small). Adapt the p-group
|
|
path (`p21/p21515`) to your own:
|
|
|
|
```bash
|
|
module use unstable && module load modbuild/2.1.3
|
|
cd /das/work/p21/p21515/MX_Pmodule/<Module>
|
|
modbuild build \
|
|
--tmpdir=/das/work/p21/p21515/.cache/Pmodules \
|
|
--distdir=/das/work/p21/p21515/.cache/Pmodules \
|
|
<version>
|
|
# verify it loads:
|
|
module use MX unstable && module load <Module>/<version>
|
|
```
|
|
|
|
If you already built without the flags, clean up what landed in the defaults:
|
|
|
|
```bash
|
|
rm -rf /var/tmp/$USER/<Module>-<version>
|
|
rm -rf ~/.cache/Pmodules/distfiles
|
|
```
|
|
|
|
- There is **no `build` subcommand**: usage is `modbuild [BUILD_SCRIPT] [options]
|
|
<version>` — `build` above is the script's filename, so run inside the module dir.
|
|
- More flags: `--prep|--configure|--compile|--install|--all` (run *up to* that
|
|
step), `--clean-install` (remove existing module first), `-f`/`--force-rebuild`,
|
|
`-j N`, `-v`, `--debug` (make the otherwise silent long steps visible).
|
|
- `curl: (23) Failure writing output to destination` during a download means
|
|
quota/disk full at the target dir, not a network error.
|
|
|
|
## Tips & tricks
|
|
|
|
1. Module names are `name/version`, never `name/name-version`.
|
|
2. New versions start `relstage: unstable`; promote to `stable` only after the
|
|
module builds AND loads. A stable module may only depend on stable modules.
|
|
3. **Contaminated environment**: previously loaded modules can leak env vars
|
|
into the build and make it fail in odd ways. Fix:
|
|
`module purge && module load modbuild/2.1.3`, then build again.
|
|
4. Hooks run with `pipefail` + `nounset`: `yes | installer` dies via SIGPIPE —
|
|
use `printf 'y\ny\n' | installer`.
|
|
5. Module-specific pitfalls live in the per-module README — read it before
|
|
building. `ccp4/README.md` is the most battle-tested and also documents the
|
|
common modbuild traps ("done" without success — no errexit in hooks,
|
|
partial downloads reused silently, slow installs on shared /opt/psi).
|
|
|
|
## Building with an AI agent
|
|
|
|
The repo ships the **new-module** skill at `.claude/skills/new-module/`
|
|
(`SKILL.md` = workflow + decision tree, `REFERENCE.md` = full schema, hooks and
|
|
copy-paste templates). It scaffolds the 4 files for a new module from the right
|
|
install-type archetype and prints the Ra build recipe.
|
|
|
|
- **Claude Code**: opens the repo → skill is auto-discovered. Type
|
|
`/new-module <tool, version, download URL, special requirements>` or simply
|
|
describe the task ("package X as a module") and it triggers.
|
|
- **Other agents** (Cursor, Codex, generic LLM chat): the skill is plain
|
|
Markdown — point the agent at `.claude/skills/new-module/SKILL.md` first,
|
|
then `REFERENCE.md`, and ask it to follow them.
|
|
- **New software (or a known tool that moved to a new download site)**: give
|
|
the agent the software's installation/download page as part of the context —
|
|
the skill knows the repo's patterns, not vendor-specific install procedures.
|
|
- **The skill can be buggy** — use it with caution and double-check what it
|
|
scaffolds before building; corrections are welcome (edit
|
|
`.claude/skills/new-module/` and commit). Reading along while it works is
|
|
also a good way to learn how these modules are put together.
|
|
- The skill copies from real modules in this repo as ground truth (`xds`,
|
|
`careless`, `DIALS`, `ccp4`, ...) — keep those exemplary, and fold new
|
|
lessons back into the skill files so the next build starts smarter.
|
|
|
|
## Pmodules background (from the wiki)
|
|
|
|
- Modules are organized in hierarchical groups; everything here is in the `MX`
|
|
group (`module use MX`).
|
|
- Filesystem layout: modulefile at `$PREFIX/GROUP/modulefiles/NAME/VERSION`,
|
|
installation at `$PREFIX/GROUP/NAME/VERSION`, per-version config at
|
|
`$PREFIX/GROUP/modulefiles/NAME/.config-VERSION` (YAML).
|
|
- Release stages: **unstable**, **stable**, **deprecated**. New builds land in
|
|
unstable; opt in with `module use unstable`.
|
|
- Pmodules automatically sets `$NAME_DIR`, `$NAME_HOME`, `$NAME_PREFIX`,
|
|
`$NAME_VERSION` on load and prepends standard directories (`$PREFIX/bin` to
|
|
`PATH`, ...) if they exist — no explicit `setenv`/`prepend-path` needed for
|
|
those.
|
|
|
|
## Permissions
|
|
|
|
Members of the `unx-mx_adm` group can update `/opt/psi/MX` directly — no -adm
|
|
account needed. Permission problems: tell the admins.
|
|
|
|
## Tutorial branches (build-*)
|
|
|
|
Each participant works on their own `build-<module>` branch (e.g.
|
|
`build-dials`), all starting from `main`.
|
|
|
|
While a branch is still identical to main (fast-forward):
|
|
|
|
```bash
|
|
git push origin main:build-dials main:build-buster ...
|
|
for b in dials buster ...; do git branch -f build-$b main; done
|
|
```
|
|
|
|
Once branches have diverged, merge main into each instead:
|
|
|
|
```bash
|
|
for b in dials buster ...; do
|
|
git checkout build-$b && git merge main && git push
|
|
done
|
|
git checkout main
|
|
```
|
|
|
|
If `git push origin main:build-xxx` fails with "non-fast-forward", the branch
|
|
has its own commits: use the merge loop, never force-push over other people's
|
|
work.
|
|
|
|
## Resources
|
|
|
|
- Buildblock examples: https://github.com/Pmodules/buildblocks/tree/master
|
|
- modbuild documentation: https://pmodules.github.io/modbuild/
|
|
- Environment Modules: https://envmodules.io/
|
|
|
|
## Acknowledgements
|
|
|
|
Many thanks to [Achim Gsell](https://www.psi.ch/en/lsm/people/achim-gsell)
|
|
for giving the MX group the tutorial on 2026-08-18, and to
|
|
[Hans Viessmann](https://www.psi.ch/en/awi/people/hans-nikolai-viessmann)
|
|
and Achim for their support.
|