= Overlays
:TOC:
:sectnums:

Please note: This document is work in progress!

== What are Overlays and for what can they be used?

== Overlay configuration

=== System wide

==== Example
.A system wide configuration file `/opt/psi/config/Pmodules.yaml`
====
....
DefaultGroups: Tools:Programming
DefaultReleaseStages: stable
ReleaseStages: unstable:stable:deprecated
TmpDir: /opt/psi/var/tmp/${USER}
DistfilesDir: /opt/psi/var/distfiles

Overlays:
  base:
    install_root: /opt/psi
    modulefiles_root: /opt/psi
  devel:
    install_root: /opt/psi
    modulefiles_root: ${HOME}/modulefiles
....
====
==== YAML Format
.Format
====
....
Overlays:
  <name>:
    install_root: <installation-root>
    modulefiles_root: <modulefiles-root>
    type: <type of overlay>
  ...
....
====
`<name>`:: Name of overlay. Note: whitespace in the name is not supported.
`<name>.install_root`:: The root of the software installation. This key is mandatory.
`<name>.modulefiles_root`:: The root of the modulefile hierarchy. This key is optional and defaults to `<name>.install_root>
`<name>.type`:: The type of the overlay, see below. This key is optional and the default value is `n`.

==== Overlay types
`n`:: Normal overlay.
`h`:: Hiding overlay.
`r`:: Replacing overlay.

=== User defined overlays

Each user can define his own overlays in `$HOME/.Pmodules/Pmodules.yaml`. 

== Working with overlays

== Building modules

The old format of the variants file is simple but very limited and almost impossible to extend for new features. To overcome the limitations a new format using YAML for variants files has been introduced. For the time being both format are supported. But it is highly recommended to use the YAML format for new modules and to migrate existing variants files in the old format to the new.

=== With a YAML variants file

Information is in the YAML configuration file:

==== General structure

name:: The name of the Pmodule without version. Example: `hdf5`
group:: group the module is in. Example: `MPI`
overlay:: (optional) overlay, defaults to `base` if not specified
relstage:: (optional) default release stage. If not defined,
 it defaults to `unstable`,
url:: (optional) download URL, Must be either define here or per
 version. The environment variables `$P`, `$V`, `$V_PKG` etc. can be used.  The downloaded file is stored with the file name `$P-${V_PKG}.ext`. Whereby `ext` is derived from the URL.
addon_sources:: (optional) array of sources for add-ons. Example: `git-lfs`
use:: (optional) use flags.
systems:: (optional) array of supported systems, defaults to any. Example: `[rhel6 rhel7]`.
compilers:: (optional) array pf supported compilers, defaults to any. Example: `[gcc intel]`.
compile-in-src:: (optional) compile in source tree, allowed vaules are `yes` and `no`, the default is `no`.
configure-with:: (optional) If a software supports CMake and autotools, this specifies the tool to use. Allowed values are `autotools`, `cmake`, `auto`, `none`. The default is `auto`.
versions:: array of `version` objects

==== Objects of type `addon_source`

name:: unique name
url:: download URL.
shasum:: (optional) SHA256 sum.

==== Objects of type `version`

dependencies:: other run-time dependencies, if any
build_with:: dependencies required for building, if any
overlay:: (optional) override the default. Example: `devel`.
relstage:: (optional) override the default. Example: `unstable`.
addon_sources:: (optional) array of sources for add-ons.
systems:: (optional) override the default
compilers:: (optional) override the default
compile-in-src:: (optional) override the default
configure-with:: (optional) override the default
variants:: array of `variant` objects

==== Object of type `variant`
with:: hierarchical dependencies, `with` must be specified only 
for hierarchical groups.
dependencies:: other run-time dependencies, if any
build_requires:: dependencies required for building, if any
addon_sources:: (optional) array of sources for add-ons.
use:: (optional) use flags
overlay:: (optional) override the default. Example: `devel`.
relstage:: (optional) override the default. Example: `unstable`.
systems:: (optional) override the default
compilers:: (optional) override the default

==== Example of a configuration files in YAML format

.YAML configuration file for serial HDF5
====
....
name: hdf5
group: MPI
overlay: base
relstage: stable
url: https://support.hdfgroup.org/ftp/HDF5/releases/$P-${V_MAJOR}.${V_MINOR}/$P-${V_PKG}/src/$P-${V_PKG}.tar.gz
systems: [rhel6,rhel7,rhel8]
compilers: [gcc,intel,pgi]
compile_in_sourcetree: no
configure_with: auto

versions:
  1.12.2:
    shasum: 3016ea56a175d2ca7f2568c8016420f7a2aad8f95e214fe7fa5485f4b80fbe51
    variants:
      - group_deps: 
          compiler: 
            gcc: [5.5.0,6.5.0,7.5.0,10.2.0,10.3.0]
            intel: ....
          mpi:
            openmpi: [...]
          #hdf5:
        runtime_deps: 
          cuda:10.0
          ...
        build_deps:
          name:version



      - with:
        - gcc: [8.5.0,9.5.0,11.3.0,12.1.0]
        relstage: unstable
        overlay: devel                 
....
====

.YAML configuration file for HDF5 (serial and parallel)
====
....
name: hdf5
overlay: base
relstage: stable
url: https://support.hdfgroup.org/ftp/HDF5/releases/$P-${V_MAJOR}.${V_MINOR}/$P-${V_PKG}/src/$P-${V_PKG}.tar.gz
systems: [rhel6 rhel7 rhel8]
compilers: [gcc intel pgi]
compile_in_sourcetree: no
configure_with: auto

versions:
  1.12.2:
    shasum: 3016ea56a175d2ca7f2568c8016420f7a2aad8f95e214fe7fa5485f4b80fbe51
    variants:
      - with:
          - gcc [5.5.0 6.5.0 7.5.0 10.2.0 10.3.0] 
        group: Compiler
      - with:
          - gcc [8.5.0,9.5.0,11.3.0,12.1.0]
          - openmpi [4.0.5-2]
        group: MPI
        relstage: unstable
        overlay: devel                 
....
====

.YAML configuration file for Git with git-lfs
====
....
name: git
defaults:
  group: Tools
  overlay: base
  relstage: stable
  url: https://mirrors.edge.kernel.org/pub/software/scm/$P/$P-${V_PKG}.tar.xz
  systems: [rhel6 rhel7 rhel8]
  compilers: [gcc]
  compile_in_sourcetree: yes
  configure_with: auto
  script: build

addons:
  git-lfs/3.2.0:
    - url: https://github.com/git-lfs/git-lfs/archive/refs/tags/v3.2.0.tar.gz
      shasum: f8e6bbe043b97db8a5c16da7289e149a3fed9f4d4f11cffcc6e517c7870cd9e5
      build_requires: [go/1.19] 
      build_script: build-git-lfs   

versions:
  2.37.2:
    shasum: 4c428908e3a2dca4174df6ef49acc995a4fdb1b45205a2c79794487a33bc06e5
    variants:
      - dependencies: TclTk/8.6.9
        build_requires: [perl/5.30.0 asciidoc/8.6.9-1 xmlto/0.0.28 gettext/0.19.8]
        relstage: unstable
        overlay: devel
        addons:
          git-lfs: 3.2.0
      
....
====

==== Format specification
.YAML format
....
name: <Pmodule-name-without-version>
source_url: "https://..."
overlay: <opt-default-overlay>
group: <default-group>
relstage: <opt-default-release-stage>
systems: [opt-array-of-supported]
compilers: [opt-array-of-supported-compilers]
compile-in-src: yes|no
configure-with: cmake|autotools|auto|none

versions:
  - <version1>:
      overlay: <overlay>
      group: <group>
      relstage: <release-stage>
      systems: [supported-systems]
      sources:
        - url: <url>
          shasum: <shasum>
      compilers: <default-list-of-supported-compilers>
      compile-in-src: yes|no
      configure-with: cmake|autotools|auto|none
      variants:
        - with:
          - gcc: [5.5.0,6.5.0,7.5.0,10.2.0,10.3.0]
          dependencies: [array-of-runtime-deps]
          build_requires: [array-of-build-deps]
          relstage: <release-stage>
          use: [array-of-use-flags]
          overlay: <overlay>
        ...
....
==== Defaults
Default values can be overriden per version/variant.

`overlay`:: The default overlay the module will be installed in. This value can be overriden for dedicated versions/variants.
`systems`:: The default for supported systems.

==== Versions and Variants

`<name>/<version>`:: An array with variants for this version.

`<name>/<version>.[i].with`:: Hierarchical dependencies for variant `i`.

`<name>/<version>.[i].dependencies`:: Build/run-time dependencies for variant `i`.

`<name>/<version>.[i].relstage`:: Relase stage of variant `i`.

`<name>/<version>.[i].overlay`:: Overlay of variant `i`.

`<name>/<version>.[i].systems`:: Supported systems.

=== Legacy format
