pmodules/Pmodules

Fork 0

mirror of https://github.com/Pmodules/Pmodules.git synced 2026-06-27 18:13:08 +02:00

Files

T

gsell 2df2e6e652 Overlay doc updaated

2022-08-22 10:33:23 +02:00

8.2 KiB

Raw Blame History

Overlays

Table of Contents

1. What are Overlays and for what can they be used?
2. Overlay configuration
- 2.1. System wide
- 2.2. User defined overlays
3. Working with overlays
4. Building modules
- 4.1. With a YAML variants file
- 4.2. Legacy format

Please note: This document is work in progress!

1. What are Overlays and for what can they be used?

2. Overlay configuration

2.1. System wide

2.1.1. Example

Example 1. A system wide configuration file /opt/psi/config/Pmodules.yaml

DefaultGroups: Tools:Programming
DefaultReleaseStages: stable
ReleaseStages: unstable:stable:deprecated
TmpDir: /opt/psi/var/tmp/${USER}
DistfilesDir: /opt/psi/var/distfiles

Overlays:
  base:
    install_root: /opt/psi
    modulefiles_root: /opt/psi
  devel:
    install_root: /opt/psi
    modulefiles_root: ${HOME}/modulefiles

2.1.2. YAML Format

Example 2. Format

Overlays:
  <name>:
    install_root: <installation-root>
    modulefiles_root: <modulefiles-root>
    type: <type of overlay>
  ...

<name>: Name of overlay. Note: whitespace in the name is not supported.
<name>.install_root: The root of the software installation. This key is mandatory.
<name>.modulefiles_root: The root of the modulefile hierarchy. This key is optional and defaults to `<name>.install_root>
<name>.type: The type of the overlay, see below. This key is optional and the default value is n.

2.1.3. Overlay types

n: Normal overlay.
h: Hiding overlay.
r: Replacing overlay.

2.2. User defined overlays

Each user can define his own overlays in $HOME/.Pmodules/Pmodules.yaml.

3. Working with overlays

4. Building modules

The old format of the variants file is simple but very limited and almost impossible to extend for new features. To overcome the limitations a new format using YAML for variants files has been introduced. For the time being both format are supported. But it is highly recommended to use the YAML format for new modules and to migrate existing variants files in the old format to the new.

4.1. With a YAML variants file

Information is in the YAML configuration file:

4.1.1. General structure

name: The name of the Pmodule without version. Example: hdf5
group: group the module is in. Example: MPI
overlay: (optional) overlay, defaults to base if not specified
relstage: (optional) default release stage. If not defined, it defaults to unstable,
url: (optional) download URL, Must be either define here or per version. The environment variables $P, $V, $V_PKG etc. can be used. The downloaded file is stored with the file name $P-${V_PKG}.ext. Whereby ext is derived from the URL.
addon_sources: (optional) array of sources for add-ons. Example: git-lfs
use: (optional) use flags.
systems: (optional) array of supported systems, defaults to any. Example: [rhel6 rhel7].
compilers: (optional) array pf supported compilers, defaults to any. Example: [gcc intel].
compile-in-src: (optional) compile in source tree, allowed vaules are yes and no, the default is no.
configure-with: (optional) If a software supports CMake and autotools, this specifies the tool to use. Allowed values are autotools, cmake, auto, none. The default is auto.
versions: array of version objects

4.1.2. Objects of type `addon_source`

name: unique name
url: download URL.
shasum: (optional) SHA256 sum.

4.1.3. Objects of type `version`

dependencies: other run-time dependencies, if any
build_with: dependencies required for building, if any
overlay: (optional) override the default. Example: devel.
relstage: (optional) override the default. Example: unstable.
addon_sources: (optional) array of sources for add-ons.
systems: (optional) override the default
compilers: (optional) override the default
compile-in-src: (optional) override the default
configure-with: (optional) override the default
variants: array of variant objects

4.1.4. Object of type `variant`

with: hierarchical dependencies, with must be specified only for hierarchical groups.
dependencies: other run-time dependencies, if any
build_requires: dependencies required for building, if any
addon_sources: (optional) array of sources for add-ons.
use: (optional) use flags
overlay: (optional) override the default. Example: devel.
relstage: (optional) override the default. Example: unstable.
systems: (optional) override the default
compilers: (optional) override the default

4.1.5. Example of a configuration files in YAML format

Example 3. YAML configuration file for serial HDF5

name: hdf5
group: MPI
overlay: base
relstage: stable
url: https://support.hdfgroup.org/ftp/HDF5/releases/$P-${V_MAJOR}.${V_MINOR}/$P-${V_PKG}/src/$P-${V_PKG}.tar.gz
systems: [rhel6,rhel7,rhel8]
compilers: [gcc,intel,pgi]
compile_in_sourcetree: no
configure_with: auto

versions:
  1.12.2:
    shasum: 3016ea56a175d2ca7f2568c8016420f7a2aad8f95e214fe7fa5485f4b80fbe51
    variants:
      - group_deps:
          compiler:
            gcc: [5.5.0,6.5.0,7.5.0,10.2.0,10.3.0]
            intel: ....
          mpi:
            openmpi: [...]
          #hdf5:
        runtime_deps:
          cuda:10.0
          ...
        build_deps:
          name:version



      - with:
        - gcc: [8.5.0,9.5.0,11.3.0,12.1.0]
        relstage: unstable
        overlay: devel

Example 4. YAML configuration file for HDF5 (serial and parallel)

name: hdf5
overlay: base
relstage: stable
url: https://support.hdfgroup.org/ftp/HDF5/releases/$P-${V_MAJOR}.${V_MINOR}/$P-${V_PKG}/src/$P-${V_PKG}.tar.gz
systems: [rhel6 rhel7 rhel8]
compilers: [gcc intel pgi]
compile_in_sourcetree: no
configure_with: auto

versions:
  1.12.2:
    shasum: 3016ea56a175d2ca7f2568c8016420f7a2aad8f95e214fe7fa5485f4b80fbe51
    variants:
      - with:
          - gcc [5.5.0 6.5.0 7.5.0 10.2.0 10.3.0]
        group: Compiler
      - with:
          - gcc [8.5.0,9.5.0,11.3.0,12.1.0]
          - openmpi [4.0.5-2]
        group: MPI
        relstage: unstable
        overlay: devel

Example 5. YAML configuration file for Git with git-lfs

name: git
defaults:
  group: Tools
  overlay: base
  relstage: stable
  url: https://mirrors.edge.kernel.org/pub/software/scm/$P/$P-${V_PKG}.tar.xz
  systems: [rhel6 rhel7 rhel8]
  compilers: [gcc]
  compile_in_sourcetree: yes
  configure_with: auto
  script: build

addons:
  git-lfs/3.2.0:
    - url: https://github.com/git-lfs/git-lfs/archive/refs/tags/v3.2.0.tar.gz
      shasum: f8e6bbe043b97db8a5c16da7289e149a3fed9f4d4f11cffcc6e517c7870cd9e5
      build_requires: [go/1.19]
      build_script: build-git-lfs

versions:
  2.37.2:
    shasum: 4c428908e3a2dca4174df6ef49acc995a4fdb1b45205a2c79794487a33bc06e5
    variants:
      - dependencies: TclTk/8.6.9
        build_requires: [perl/5.30.0 asciidoc/8.6.9-1 xmlto/0.0.28 gettext/0.19.8]
        relstage: unstable
        overlay: devel
        addons:
          git-lfs: 3.2.0

4.1.6. Format specification

YAML format

name: <Pmodule-name-without-version>
source_url: "https://..."
overlay: <opt-default-overlay>
group: <default-group>
relstage: <opt-default-release-stage>
systems: [opt-array-of-supported]
compilers: [opt-array-of-supported-compilers]
compile-in-src: yes|no
configure-with: cmake|autotools|auto|none

versions:
  - <version1>:
      overlay: <overlay>
      group: <group>
      relstage: <release-stage>
      systems: [supported-systems]
      sources:
        - url: <url>
          shasum: <shasum>
      compilers: <default-list-of-supported-compilers>
      compile-in-src: yes|no
      configure-with: cmake|autotools|auto|none
      variants:
        - with:
          - gcc: [5.5.0,6.5.0,7.5.0,10.2.0,10.3.0]
          dependencies: [array-of-runtime-deps]
          build_requires: [array-of-build-deps]
          relstage: <release-stage>
          use: [array-of-use-flags]
          overlay: <overlay>
        ...

4.1.7. Defaults

Default values can be overriden per version/variant.

overlay: The default overlay the module will be installed in. This value can be overriden for dedicated versions/variants.
systems: The default for supported systems.

4.1.8. Versions and Variants

<name>/<version>: An array with variants for this version.
<name>/<version>.[i].with: Hierarchical dependencies for variant i.
<name>/<version>.[i].dependencies: Build/run-time dependencies for variant i.
<name>/<version>.[i].relstage: Relase stage of variant i.
<name>/<version>.[i].overlay: Overlay of variant i.
<name>/<version>.[i].systems: Supported systems.

8.2 KiB Raw Blame History