From c3c38856b329d6315d9c2b0db98451a7b7f30484 Mon Sep 17 00:00:00 2001 From: Dawn Date: Tue, 18 Aug 2026 22:18:37 +0200 Subject: [PATCH] Docs overhaul: quickstart, current commands everywhere, credits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Main README: clone/SSH/branching quickstart with full /das paths, --tmpdir/--distdir as part of the standard build command (do not fill /var/tmp or $HOME) with cleanup for defaults, tips deduplicated against module READMEs, old-system references removed, resource links and tutorial acknowledgements added. Module READMEs: purge outdated recipes repo-wide — Pmodules/1.1.20 and 2.0.0 plus ./build replaced by modbuild/2.1.3 invocations; the cctbx unbound-variable fix moved from the main README into phenix and DIALS READMEs where it belongs. Co-Authored-By: Claude Fable 5 --- DIALS/README.md | 10 ++-- README.md | 102 +++++++++++++++++++++++++--------------- Xtrapol8/README.md | 2 +- alphafold/README.md | 2 +- autoPROC/README.md | 6 +-- careless/README.md | 2 +- crystfel/README.md | 4 +- jfjoch_viewer/README.md | 6 +-- phenix/README.md | 8 ++++ pymol/README.md | 4 +- xds/README.md | 6 +-- xdsgui/README.md | 2 +- 12 files changed, 96 insertions(+), 58 deletions(-) diff --git a/DIALS/README.md b/DIALS/README.md index 3216c6c..04cbe11 100644 --- a/DIALS/README.md +++ b/DIALS/README.md @@ -2,11 +2,11 @@ Step 1: make sure you are in the `unx-mx_adm` group -- no -adm account needed, members can update the modules in `/opt/psi/MX` directly (if you have problems, please let the admins know) -Step 2: load the latest Pmodules package +Step 2: load the build tool (modbuild) ``` module use unstable -module load Pmodules/2.0.0 +module load modbuild/2.1.3 ``` Step 3: go into DIALS directory, update config.yaml to add newest release version and build module with that release name @@ -14,7 +14,7 @@ Step 3: go into DIALS directory, update config.yaml to add newest release versio ``` cd DIALS vi files/config.yaml -./build 3.25.0 +modbuild build 3.25.0 ``` Step 4: confirm that the module loads as expected @@ -23,3 +23,7 @@ PS: Unsupported dtype issue is resolved by changing modules/dxtbx/src/dxtbx/format/FormatNXmx.py allowing the return of also /entry/data/data to the handle: https://github.com/dials/dials/issues/3033 + +If loading the module fails with an `unbound variable` from the sourced +`dials_env.sh` (cctbx-style env script), add `puts stdout "set +x nounset"` +before the `source` line in the modulefile — same fix as phenix. diff --git a/README.md b/README.md index 08e63d2..ebc401d 100644 --- a/README.md +++ b/README.md @@ -14,34 +14,62 @@ exactly 4 files: Modules build on the **Ra** cluster (RHEL8) into `/opt/psi/MX//`. Upstream docs: https://github.com/Pmodules/Pmodules/wiki +## Getting the repo + +Clone into your work folder on Ra (once). SSH access to Gitea is recommended: +generate a key ([how-to](https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key)) +and add the public key at https://gitea.psi.ch/user/settings/keys. + +```bash +cd /das/work/p21/p21515/ # adapt the p-group path to your own +git clone git@gitea.psi.ch:mx/MX_Pmodule.git +cd /das/work/p21/p21515/MX_Pmodule +``` + +Work on your own branch, not on main: + +```bash +git switch main && git pull # start from up-to-date main +git switch -c build- # e.g. build-dials +git push -u origin build- +``` + +New to git? Start here: https://git-scm.com/doc + ## Building on Ra +`` below is one of the top-level directories of the cloned repo +(e.g. `/das/work/p21/p21515/MX_Pmodule/ccp4`): + +Always pass `--tmpdir` and `--distdir` pointing to your p-group work folder — +the defaults are `/var/tmp/$USER` (shared with everyone on the login node — +**do NOT fill it**) and `~/.cache/Pmodules/distfiles` (home quota is too +small). Adapt the p-group path (`p21/p21515`) to your own: + ```bash module use unstable && module load modbuild/2.1.3 -cd -modbuild build +cd /das/work/p21/p21515/MX_Pmodule/ +modbuild build \ + --tmpdir=/das/work/p21/p21515/.cache/Pmodules \ + --distdir=/das/work/p21/p21515/.cache/Pmodules \ + # verify it loads: module use MX unstable && module load / ``` +If you already built without the flags, clean up what landed in the defaults: + +```bash +rm -rf /var/tmp/$USER/- +rm -rf ~/.cache/Pmodules/distfiles +``` + - There is **no `build` subcommand**: usage is `modbuild [BUILD_SCRIPT] [options] ` — `build` above is the script's filename, so run inside the module dir. -- Flags: `--prep|--configure|--compile|--install|--all` (run *up to* that step), - `--clean-install` (remove existing module first), `-f`/`--force-rebuild`, `-j N`, - `--tmpdir=DIR`, `--distdir=DIR`, `-v`, `--debug`. - There is **no `--dry-run` and no `--check-mode`**. -- Big downloads/builds: the defaults `/var/tmp/$USER` (tmp) and - `~/.cache/Pmodules/distfiles` (downloads) are too small — put both on /das, - **adapting the p-group path to your own**: - - ```bash - modbuild build \ - --tmpdir=/das/work/p21/p21515/.cache/Pmodules \ - --distdir=/das/work/p21/p21515/.cache/Pmodules \ - --clean-install -v --debug - ``` - - `curl: (23) Failure writing output to destination` during a download means +- More flags: `--prep|--configure|--compile|--install|--all` (run *up to* that + step), `--clean-install` (remove existing module first), `-f`/`--force-rebuild`, + `-j N`, `-v`, `--debug` (make the otherwise silent long steps visible). +- `curl: (23) Failure writing output to destination` during a download means quota/disk full at the target dir, not a network error. ## Tips & tricks @@ -52,27 +80,12 @@ module use MX unstable && module load / 3. **Contaminated environment**: previously loaded modules can leak env vars into the build and make it fail in odd ways. Fix: `module purge && module load modbuild/2.1.3`, then build again. -4. **"done" does not mean success**: modbuild has no errexit in build hooks — - failed commands scroll by and the build still finishes. Guard critical steps - with `|| std::die 42 "message"` and end `install` by checking the file your - modulefile needs actually exists. -5. Hooks run with `pipefail` + `nounset`: `yes | installer` dies via SIGPIPE — +4. Hooks run with `pipefail` + `nounset`: `yes | installer` dies via SIGPIPE — use `printf 'y\ny\n' | installer`. -6. prep reuses any file already in the distfiles dir with **no integrity - check** — an interrupted download leaves a partial tarball that later fails - with `gzip: unexpected end of file`. Either `rm` it and re-run, pin - `shasums:`, or use a custom prep with `wget --continue` + `gzip -t` - (see `ccp4/build`). -7. Unbound-variable on load (cctbx/phenix-style env scripts): add - `puts stdout "set +x nounset"` before the `source` line in the modulefile. -8. Do not use `pbuild::add_to_group`, `use_cmake`, `use_autotools`, - `set_download_url` — removed, they `std::die`. Use `group:`, `urls:`, - `configure_with:` in config.yaml instead. -9. `/opt/psi` is shared storage: untarring/installing GBs runs 10–30+ min with - no output. Watch `du -sh /opt/psi/MX//` from a second shell - instead of killing a healthy build. -10. Check the per-module README before building — module-specific pitfalls are - documented there (`ccp4/README.md` is the most battle-tested example). +5. Module-specific pitfalls live in the per-module README — read it before + building. `ccp4/README.md` is the most battle-tested and also documents the + common modbuild traps ("done" without success — no errexit in hooks, + partial downloads reused silently, slow installs on shared /opt/psi). ## Building with an AI agent @@ -134,3 +147,16 @@ git checkout main If `git push origin main:build-xxx` fails with "non-fast-forward", the branch has its own commits: use the merge loop, never force-push over other people's work. + +## Resources + +- Buildblock examples: https://github.com/Pmodules/buildblocks/tree/master +- modbuild documentation: https://pmodules.github.io/modbuild/ +- Environment Modules: https://envmodules.io/ + +## Acknowledgements + +Many thanks to [Achim Gsell](https://www.psi.ch/en/lsm/people/achim-gsell) +for giving the MX group the tutorial on 2026-08-18, and to +[Hans Viessmann](https://www.psi.ch/en/awi/people/hans-nikolai-viessmann) +and Achim for their support. diff --git a/Xtrapol8/README.md b/Xtrapol8/README.md index 3ddc946..bd36547 100644 --- a/Xtrapol8/README.md +++ b/Xtrapol8/README.md @@ -21,7 +21,7 @@ versions (tagged `v1.x.y`) would need a per-version `urls:` override. ## Building on Ra ```sh -module use unstable && module load modbuild/2.1.2 +module use unstable && module load modbuild/2.1.3 cd Xtrapol8 modbuild build 1.2.9 module use MX unstable && module load Xtrapol8/1.2.9 diff --git a/alphafold/README.md b/alphafold/README.md index 7e82f71..a0bb3e2 100644 --- a/alphafold/README.md +++ b/alphafold/README.md @@ -97,7 +97,7 @@ Add version to files/variants. The version number should match a github tag As a member of the `unx-mx_adm` group (no -adm account needed; if you have problems, please let the admins know): ``` cd alphafold -./build +modbuild build ``` ## Testing diff --git a/autoPROC/README.md b/autoPROC/README.md index dc57496..325a63d 100644 --- a/autoPROC/README.md +++ b/autoPROC/README.md @@ -10,11 +10,11 @@ dependencies. Step 1: make sure you are in the `unx-mx_adm` group -- no -adm account needed, members can update the modules in `/opt/psi/MX` directly (if you have problems, please let the admins know) -Step 2: load the latest Pmodules package +Step 2: load the build tool (modbuild) ``` module use unstable -module load Pmodules/2.0.0 +module load modbuild/2.1.3 ``` Step 3: stage the licence-gated snapshot. Download it from Global Phasing (needs @@ -38,7 +38,7 @@ vi files/config.yaml Step 5: build (installs to `/opt/psi/MX/autoPROC/`) ``` -./build 20240710 +modbuild build 20240710 ``` Step 6: confirm the module loads and autoPROC is licensed diff --git a/careless/README.md b/careless/README.md index ff05205..43a574e 100644 --- a/careless/README.md +++ b/careless/README.md @@ -58,4 +58,4 @@ careless mono dHKL /tmp/careless_test --iterations 10 ## Adding a new version Add the version under `versions:` in `files/config.yaml`, then on a Ra -login node: `modbuild build ` (modbuild/2.1.2). +login node: `modbuild build ` (modbuild/2.1.3). diff --git a/crystfel/README.md b/crystfel/README.md index f39f7cc..d6674ae 100644 --- a/crystfel/README.md +++ b/crystfel/README.md @@ -24,7 +24,7 @@ Loading the module puts `$PREFIX/bin` on `PATH`. ## Building on Ra ```sh -module use unstable && module load modbuild/2.1.2 +module use unstable && module load modbuild/2.1.3 cd crystfel modbuild build 0.13.0 module use MX unstable && module load crystfel/0.13.0 && indexamajig --version @@ -43,4 +43,4 @@ cell_explorer # GUI check, needs X Add the version under `versions:` and its tarball sha256 under `shasums:` in `files/config.yaml`, then on a Ra login node: -`modbuild build ` (modbuild/2.1.2). +`modbuild build ` (modbuild/2.1.3). diff --git a/jfjoch_viewer/README.md b/jfjoch_viewer/README.md index 0c4fed2..4fb3fbe 100644 --- a/jfjoch_viewer/README.md +++ b/jfjoch_viewer/README.md @@ -2,11 +2,11 @@ Step 1: make sure you are in the `unx-mx_adm` group -- no -adm account needed, members can update the modules in `/opt/psi/MX` directly (if you have problems, please let the admins know) -Step 2: load the latest Pmodules package +Step 2: load the build tool (modbuild) ``` module use unstable -module load Pmodules/1.1.20 +module load modbuild/2.1.3 ``` Step 3: go into jfjoch_viewer directory, update config.yaml to add newest release version and build module with that release name @@ -14,7 +14,7 @@ Step 3: go into jfjoch_viewer directory, update config.yaml to add newest releas ``` cd jfjoch_viewer vi files/config.yaml -./build 1.0.0-rc.156 +modbuild build 1.0.0-rc.156 ``` Step 4: confirm that the module loads as expected diff --git a/phenix/README.md b/phenix/README.md index 8db5c37..c19c754 100644 --- a/phenix/README.md +++ b/phenix/README.md @@ -21,6 +21,14 @@ rosetta.build_phenix_interface nproc=10 - run rosetta.run_tests to test it. +### Unbound variable on module load + +Sourcing the phenix env script can fail under `nounset`, e.g. +`phenix/phenix-1.20-4459/build/setpaths.sh: line 4: +LIBTBX_BUILD_RELOCATION_HINT: unbound variable`. Fix: add +`puts stdout "set +x nounset"` before the `source` line in the +modulefile. (Applies to any cctbx-style env script — see also DIALS.) + ### Additional Remarks the phenix help pages (BB archive) does not work. diff --git a/pymol/README.md b/pymol/README.md index d78029a..6b78e5c 100644 --- a/pymol/README.md +++ b/pymol/README.md @@ -30,7 +30,7 @@ prompts on first start (get the file from https://pymol.org/dsc/ip/license/). ## Building on Ra ```sh -module use unstable && module load modbuild/2.1.2 +module use unstable && module load modbuild/2.1.3 cd pymol modbuild build 3.1.8 module use MX unstable && module load pymol/3.1.8 @@ -49,4 +49,4 @@ pymol # GUI check, needs X Check the exact bundle filename on https://pymol.org (the `_0` build number and `py310` tag change between releases), update `urls:` if needed, add the version + tarball sha256 in `files/config.yaml`, then -`modbuild build ` (modbuild/2.1.2) on a Ra login node. +`modbuild build ` (modbuild/2.1.3) on a Ra login node. diff --git a/xds/README.md b/xds/README.md index a491991..52347d3 100644 --- a/xds/README.md +++ b/xds/README.md @@ -2,11 +2,11 @@ Step 1: make sure you are in the `unx-mx_adm` group -- no -adm account needed, members can update the modules in `/opt/psi/MX` directly (if you have problems, please let the admins know) -Step 2: load the latest Pmodules package +Step 2: load the build tool (modbuild) ``` module use unstable -module load Pmodules/1.1.20 +module load modbuild/2.1.3 ``` Step 3: go into xds directory, update config.yaml to add newest release version and build module with that release name (e.g. below is for 20240724) @@ -14,7 +14,7 @@ Step 3: go into xds directory, update config.yaml to add newest release version ``` cd xds vi files/config.yaml -./build 20240724 +modbuild build 20240724 ``` Step 4: confirm that the module loads as expected \ No newline at end of file diff --git a/xdsgui/README.md b/xdsgui/README.md index fe02b46..f9b0788 100644 --- a/xdsgui/README.md +++ b/xdsgui/README.md @@ -31,4 +31,4 @@ xdsgui # on a node with X forwarding; GUI opens ## Adding a new version Add the new download date under `versions:` in `files/config.yaml`, -then on a Ra login node: `modbuild build ` (modbuild/2.1.2). +then on a Ra login node: `modbuild build ` (modbuild/2.1.3).