The PSI Spack Deployment
How to use on Merlin systems (Merlin6/Merlin7/Ra)
Load the spack module
[stable] user@login001:~> module load spack
Voilà you're done you can use Spack! :)
- Side note: the default software stack loaded for you is stable, which might not contain all the packages you're looking for,
so if you want to use the unstable software stack switch to it using:
spack env activate -p unstable
Find out what's already installed
[unstable] user@login001:~> spack find
==> In environment unstable
==> 1 root specs
- []
==> Included specs
-- no arch / gcc ------------------------------------------------
gromacs%gcc
-- no arch / gcc@12.3.0 -----------------------------------------
cp2k@2024.1%gcc@12.3.0 opal@master%gcc@12.3.0 py-alphafold@2.3.2%gcc@12.3.0
cp2k@2024.1%gcc@12.3.0 py-alphafold@2.3.2%gcc@12.3.0 quantum-espresso@7.3.1%gcc@12.3.0
-- no arch / oneapi@2024.1.0 ------------------------------------
gromacs@2024.1%oneapi@2024.1.0
==> Installed packages
-- linux-rhel7-x86_64 / gcc@4.8.5 -------------------------------
autoconf@2.72 curl@8.7.1 gdbm@1.23 libffi@3.4.6 mpfr@4.2.1 pkg-config@0.29.2 readline@8.2
autoconf@2.72 diffutils@3.10 gettext@0.19.8.1 libidn2@2.3.7 mpfr@4.2.1 pkg-config@0.29.2 sqlite@3.43.2
autoconf-archive@2023.02.20 diffutils@3.10 gettext@0.19.8.1 libmd@1.0.4 ncurses@6.5 py-fypp@3.1 texinfo@7.0.3
autoconf-archive@2023.02.20 expat@2.6.2 git@2.42.0 libpciaccess@0.17 ncurses@6.5 py-fypp@3.1 util-linux-uuid@2.36.2
automake@1.16.5 findutils@4.9.0 glibc@2.17 libsigsegv@2.14 nghttp2@1.57.0 py-pip@23.1.2 util-macros@1.19.3
automake@1.16.5 findutils@4.9.0 glibc@2.17 libsigsegv@2.14 ninja@1.11.1 py-setuptools@59.4.0 xz@5.4.6
berkeley-db@18.1.40 flex@2.6.3 gmake@4.4.1 libtool@2.4.7 numactl@2.0.14 py-wheel@0.41.2 zlib-ng@2.1.6
berkeley-db@18.1.40 gawk@5.3.0 gmake@4.4.1 libunistring@1.2 nvhpc@24.3 python@3.8.18 zlib-ng@2.1.6
binutils@2.42 gawk@5.3.0 gmp@6.2.1 libxcrypt@4.4.35 openssh@9.7p1 python@3.9.18 zstd@1.5.6
binutils@2.42 gcc@12.3.0 gmp@6.2.1 libxml2@2.10.3 openssl@3.3.0 python@3.11.7 zstd@1.5.6
binutils@2.42 gcc@12.3.0 krb5@1.20.1 m4@1.4.19 openssl@3.3.0 python-venv@1.0
bison@3.8.2 gcc-runtime@4.8.5 libbsd@0.12.1 m4@1.4.19 pcre2@10.43 python-venv@1.0
bzip2@1.0.8 gcc-runtime@4.8.5 libedit@3.1-20230828 mpc@1.3.1 perl@5.38.0 re2c@2.2
bzip2@1.0.8 gdbm@1.23 libevent@2.1.12 mpc@1.3.1 perl@5.38.0 readline@8.2
-- linux-rhel7-x86_64 / gcc@12.3.0 ------------------------------
abseil-cpp@20240116.2 libogg@1.3.5 py-beniget@0.3.0 py-ml-collections@0.1.0 py-tabulate@0.8.9
amrex@18.07 libpng@1.6.39 py-biopython@1.79 py-networkx@2.7.1 py-tensorboard@2.11.2
aria2@1.37.0 libssh2@1.11.0 py-bottleneck@1.3.2 py-numexpr@2.8.4 py-tensorboard-data-server@0.6.1
boost@1.85.0 libtheora@1.1.1 py-cachetools@5.2.0 py-numpy@1.17.5 py-tensorboard-plugin-wit@1.8.1
boost@1.85.0 libxc@6.2.2 py-certifi@2023.7.22 py-numpy@1.21.6 py-tensorflow@2.11.0
c-ares@1.27.0 libxsmm@1.17 py-charset-normalizer@3.3.0 py-oauthlib@3.2.2 py-termcolor@1.1.0
cmake@3.27.9 libyaml@0.2.5 py-chex@0.0.7 py-opt-einsum@3.3.0 py-tomli@2.0.1
cp2k@2024.1 llvm@16.0.6 py-contextlib2@21.6.0 py-packaging@23.1 py-toolz@0.12.0
cp2k@2024.1 lua@5.3.6 py-cython@0.29.36 py-pandas@1.3.4 py-typing-extensions@4.8.0
cuda@12.4.0 lz4@1.9.4 py-decorator@5.1.1 py-pdbfixer@1.7 py-urllib3@1.26.12
fftw@3.3.10 mesa-glu@9.0.2 py-dm-haiku@0.0.9 py-pip@23.0 py-websocket-client@1.6.3
fftw@3.3.10 mesa18@18.3.6 py-dm-tree@0.1.6 py-pip@23.1.2 py-werkzeug@2.0.2
freetype@2.10.2 metis@5.1.0 py-docker@5.0.3 py-ply@3.11 py-wheel@0.37.1
gcc-runtime@12.3.0 mithra@2.0 py-etils@0.9.0 py-poetry-core@1.8.1 py-wheel@0.41.2
gcc-runtime@12.3.0 netlib-scalapack@2.2.0 py-flatbuffers@23.5.26 py-protobuf@3.19.4 py-wrapt@1.15.0
gettext@0.19.8.1 opal@master py-flit-core@3.9.0 py-pyasn1@0.4.8 py-zipp@3.17.0
gl2ps@1.4.2 openblas@0.3.25 py-gast@0.4.0 py-pyasn1-modules@0.2.8 python@3.8.18
googletest@1.12.1 openblas@0.3.26 py-google-auth@2.27.0 py-pybind11@2.12.0 python@3.11.7
gsl@2.7.1 openblas@0.3.26 py-google-auth-oauthlib@0.4.6 py-pytest-runner@6.0.0 python-venv@1.0
h5hut@2.0.0rc6 openjdk@11.0.20.1_1 py-google-pasta@0.2.0 py-python-dateutil@2.8.2 python-venv@1.0
hdf5@1.12.3 openmm@7.5.1 py-grpcio@1.60.1 py-pythran@0.9.11 quantum-espresso@7.3.1
hdf5@1.14.3 openmpi@4.1.6 py-h5py@3.11.0 py-pytz@2023.3 re2@2023-09-01
hdf5@1.14.3 openmpi@4.1.6 py-idna@3.4 py-pyyaml@6.0 rust-bootstrap@1.78.0
hh-suite@3.3.0 osmesa@11.2.0 py-immutabledict@2.2.1 py-requests@2.31.0 slurm@23-11-0-1
hmmer@3.4 parmetis@4.0.3 py-importlib-resources@5.12.0 py-requests-oauthlib@1.3.1 swig@4.1.1
hwloc@2.9.1 pcre@8.45 py-jax@0.3.25 py-rsa@4.9 trilinos@13.4.0
jsoncpp@1.9.5 perl-data-dumper@2.173 py-jaxlib@0.3.25 py-scipy@1.7.0 unzip@6.0
kalign@3.4.0 protobuf@3.19.4 py-jmp@0.0.2 py-setuptools@57.4.0 zip@3.0
libgcrypt@1.10.3 py-absl-py@1.0.0 py-libclang@16.0.0 py-setuptools@69.2.0
libgpg-error@1.49 py-alphafold@2.3.2 py-mako@1.2.4 py-setuptools@69.2.0
libint@2.6.0 py-alphafold@2.3.2 py-markdown@3.3.4 py-setuptools-scm@8.0.4
libjpeg@9f py-astunparse@1.6.3 py-markupsafe@2.1.3 py-six@1.16.0
-- linux-rhel7-x86_64 / oneapi@2024.1.0 -------------------------
gromacs@2024.1 hwloc@2.9.1 intel-oneapi-mkl@2024.0.0 intel-oneapi-runtime@2024.1.0 intel-tbb@2021.9.0 openmpi@4.1.6 slurm@23-11-0-1
==> 257 installed packages
The included specs and installed packages sections shows you all the packages that are pre-installed for you.
In case of a package having multiple installations, the following command helps you to find out which software you're interested about.
For example let's say you are interested in py-alphafold but only in the gpu version; do:
[unstable] user@login001:~> spack find -vl py-alphafold
==> In environment unstable
==> 1 root specs
- dpjaszc visit
==> Included specs
-- no arch / gcc ------------------------------------------------
------- gromacs%gcc
-- no arch / gcc@12.3.0 -----------------------------------------
------- cp2k@2024.1%gcc@12.3.0 ------- opal@master%gcc@12.3.0 ------- py-alphafold@2.3.2%gcc@12.3.0
------- cp2k@2024.1%gcc@12.3.0 ------- py-alphafold@2.3.2%gcc@12.3.0 ------- quantum-espresso@7.3.1%gcc@12.3.0
-- no arch / oneapi@2024.1.0 ------------------------------------
------- gromacs@2024.1%oneapi@2024.1.0
==> Installed packages
-- linux-rhel7-x86_64 / gcc@12.3.0 ------------------------------
rrfdppp py-alphafold@2.3.2~cuda build_system=python_pip tqziprv py-alphafold@2.3.2+cuda build_system=python_pip cuda_arch=60
==> 2 installed packages
As you can see there are two installation of py-alphafold, one with +cuda and one without.
In this particular example, there is only one variant that interests you, so you can do:
spack load py-alphafold +cuda
However if multiple variants are interesting to you and you don't want to have to cite all of them you can do:
spack load /tqziprv # refer to the installation with its hash directly.
Install your own software
- Check if your package is already implemented in Spack.
You can also use the following website: https://packages.spack.io/
spack list $pkg_name # e.g opal
- Check the package variants you want to set e.g +cuda +openmp ^openmpi ...
spack info $pkg_name # e.g opal
- Add your package to your personal environment
spack add $pkg_name@version +variant1 ^dep1@version1
- If you want to install from local source clone your source under
/afs/psi.ch/sys/spack/user/$USER/spack-environment/$pkg_name
and tell Spack you want to use it using:
spack develop $pkg_name@version # Skip this if you don't want to develop from local source
- Check the whole dependency tree of your spec and
if you don't need to set more variants
spack concretize
- Install the packages in your environment
spack install # -v for verbose
- Load your package
spack load $pkg_name@version +variant1 ^dep1@version1
- Submit your script
sbatch batch.script
Switching between environments
There are two environment available to users; stable and unstable.
The first one is loaded by default when loading the module.
If you want to switch between the two and use different software stack you can use the following command:
spack env activate -p unstable # or stable
Managing your environments
When adding the following command you actually add a package to your own software stack;
spack add $pkg_name@version +variant1 ^dep1@version1
You can check which package will be concretized and are in your environment using;
[unstable] user@login001:~> spack find
==> In environment unstable
==> 1 root specs
- visit ~adios2+vtkm+gui
==> Included specs
-- no arch / gcc ------------------------------------------------
gromacs%gcc
-- no arch / gcc@12.3.0 -----------------------------------------
cp2k@2024.1%gcc@12.3.0 opal@master%gcc@12.3.0 py-alphafold@2.3.2%gcc@12.3.0
cp2k@2024.1%gcc@12.3.0 py-alphafold@2.3.2%gcc@12.3.0 quantum-espresso@7.3.1%gcc@12.3.0
-- no arch / oneapi@2024.1.0 ------------------------------------
gromacs@2024.1%oneapi@2024.1.0
==> Installed packages
...
==> 257 installed packages
The root spec is the specs that are personal to you and that will be concretized and installed.
The installed packages are either packages that were previously installed by admins or packages that you already installed but that are not root.
If you want to remove this package from your personal env, you can use the following command:
spack rm $pkg_name@version +variant1 ^dep1@version1
or
[stable] user@login001:~> spack config edit
spack:
...
specs:
-- - visit~adios2+gui+vtkm ^harfbuzz%gcc@12.3.0
++ []
...
and remove it manually from the spec list.
How to use on other systems
Install Spack
user@supersystem:~> cd /scratch/$USER
user@supersystem:~> git clone -c feature.manyFiles=true https://github.com/spack/spack.git
user@supersystem:~> . spack/share/spack/setup-env.sh
# Add PSI specific recipes
user@supersystem:~> git clone https://gitlab.psi.ch/lsm-hpce/alps/spack-psi.git
user@supersystem:~> spack repo add $PWD/spack-psi
user@supersystem:~> spack install $pkg_name # -v for verbose