#+TITLE Project Environment Modules #+TODO: TODO(t) IN-PROGRESS(p) WAITING(w) | DONE(d) STOPPED(s) # Environment_Modules_Project.org * Request for Comments: Environment Modules at PSI Achim Gsell, Valeri MArkushin, Hans Christian Stadler Scientific Computing, AIT, PSI 2014-04-29 * Introduction This document describes the requirements for a new implementation of *environment modules* at PSI that should address known problems with the present implementation based on the classical TCL modules [[[env_modules_tcl]]]. A detailed overview of suggested solutions was presented in the talk [[[env_modules_psi]]] by Achim. The details of the specification are described in Section [[[Specification]]]. The draft of design is presented in Section [[[Design]]]. There are two readily available candidates to meet the requrements of this project: the extended version of TCL modules [[[env_modules_psi]]] and the Lua based Lmod [[[lmod]]]. The implementation proposal is summarized in Section [[[Implementation]]]. The implementation detailes will be written when the specifications are fixed and a genelal agreement concerning the design is reached. The tecnical details concerning building specific software packages are mostly beyond the scope of this document: here we are interested only in the minimum and complete set of environment variables and dependencies required to provide and use a particular package. Our goal is to provide the specifications and design that make the job of maintaners easier and staightforward. * IN-PROGRESS Specification <> Since we have a work in progress, we call it *specification*, not evaluation. ** IN-PROGRESS Must Preserve the Core Functionality of Traditional Modules :OK: The essential (to be specified) tradinional *module* *subcommands* *must* be supported. Both TCL and Lua modules must meet this requirement. It is desirable to support all traditional subcommands. *** IN-PROGRESS The TCL module subcommands #+BEGIN_EXAMPLE module help Modules Release 3.2.10 2012-12-21 (Copyright GNU GPL v2 1991): Usage: module [ switches ] [ subcommand ] [subcommand-args ] Switches: -H|--help this usage info -V|--version modules version & configuration options -f|--force force active dependency resolution -t|--terse terse format avail and list format -l|--long long format avail and list format -h|--human readable format avail and list format -v|--verbose enable verbose messages -s|--silent disable verbose messages -c|--create create caches for avail and apropos -i|--icase case insensitive -u|--userlvl set user level to (nov[ice],exp[ert],adv[anced]) Available SubCommands and Args: + add|load modulefile [modulefile ...] + rm|unload modulefile [modulefile ...] + switch|swap [modulefile1] modulefile2 + display|show modulefile [modulefile ...] + avail [modulefile [modulefile ...]] + use [-a|--append] dir [dir ...] + unuse dir [dir ...] + update + refresh + purge + list + clear + help [modulefile [modulefile ...]] + whatis [modulefile [modulefile ...]] + apropos|keyword string + initadd modulefile [modulefile ...] + initprepend modulefile [modulefile ...] + initrm modulefile [modulefile ...] + initswitch modulefile1 modulefile2 + initlist + initclear #+END_EXAMPLE *** IN-PROGRESS The Lmod module subcommands #+BEGIN_EXAMPLE #+END_EXAMPLE ** DONE Must Control the Availability of Modules Subject to Current State and Dependencies CLOSED: [2014-04-28 Mon 10:49] The *Module Hierarchy* section has been moved to [[[DesignProposal]]]. *** DONE Must support SW Dependencies CLOSED: [2014-04-28 Mon 10:46] *** DONE Desirable to Support HW and OS Constaints CLOSED: [2014-04-28 Mon 10:47] *** DONE Desirable to Support Licensing Constaints CLOSED: [2014-04-28 Mon 10:48] Only nonfloating licenses will be taken into account. ** IN-PROGRESS Settings for Standard and Application Specific Environment Variables *** DONE Must Export a Predefined Subset of POSIX Standard and Other Common Environment Variables CLOSED: [2014-04-28 Mon 10:55] The following environment variables *must be set* by all modules: #+BEGIN_EXAMPLE _VERSION _PREFIX _DIR _HOME #+END_EXAMPLE Every module that *has effect* on the following POSIX standard environment variables (FIXME) *must explicitly export* them: #+BEGIN_EXAMPLE PATH LD_LIBRARY_PATH MANPATH #+END_EXAMPLE See [[[Open_Group_Base_Specifications_EM]]] for the list of variables that are frequently exported by widely used command interpreters and applications. In particular, the following variables are considered as *widely used*: #+BEGIN_EXAMPLE CC CFLAGS FC FFLAGS LFLAGS MAKEFLAGS MANPATH PATH #+END_EXAMPLE FIXME (non-POSIX, e.g. common GCC shared by many other applications) if corresponding path exists: #+BEGIN_EXAMPLE CPATH C_INCLUDE_PATH CPLUS_INCLUDE_PATH _INCLUDE_DIR LIBRARY_PATH _LIBRARY_DIR LD_LIBRARY_PATH #+END_EXAMPLE *** IN-PROGRESS Must Export Family Environment Variables <> #+BEGIN_EXAMPLE = _VERSION= _DIR= #+END_EXAMPLE Example: #+BEGIN_EXAMPLE MPI=openmpi MPI_VERSION=1.8.0 #+END_EXAMPLE This would replace [[[ModuleIdentifiers]]] if the following variables are used: #+BEGIN_EXAMPLE COMPILER COMPILER_VERSION COMPILER_DIR MPI MPI_VERSION MPI_DIR #+END_EXAMPLE *** IN-PROGRESS Should Export Module Identifiers <> This may be merged with [[[Family]]]. The goal is to support generic module configuration and scripts. #+BEGIN_EXAMPLE PSI_COMPILER PSI_COMPILER_VERSION PSI_COMPILER_DIR PSI_MPI PSI_MPI_VERSION PSI_MPI_DIR #+END_EXAMPLE **** IN-PROGRESS Must Export Environment Variables for Compiler Implementations #+BEGIN_EXAMPLE CC CXX F77 F90 FC FORTRAN #+END_EXAMPLE **** IN-PROGRESS Must Export Environment Variables for MPI Implementations The environment variables for MPI wrappers begin with MPI: #+BEGIN_EXAMPLE MPICC MPICXX MPIF77 MPIF90 MPIFC MPIFORTRAN MPIEXEC MPIRUN #+END_EXAMPLE OR/AND FIXME The environment variables for MPI wrappers begin with MPI_ [[[ANL_MPICH2]]]: #+BEGIN_EXAMPLE MPI_CC MPI_CXX MPI_F77 MPI_F90 MPI_FC MPI_FORTRAN MPI_EXEC MPI_RUN MPI_INC MPI_LIBS #+END_EXAMPLE Note by Achim: names without underscore seems to be more common and more commonly uses be autotools and cmake. |-------------------+---------------+---------------| | Search term | "MPICC=mpicc" | "MPICC=mpicc" | |-------------------+---------------+---------------| | Nunber of results | 4490 | 922 | | Reference | | [[[ANL_MPICH2]]] | |-------------------+---------------+---------------| The environment variables should allow to distiguish between MPI wrappers (e.g. MPICC) and ordinary compilers (e.g. CC). *** IN-PROGRESS Should Export Package- or Vendor-Defined Environment Variables Each module should set the environment variables that make its use easier for the user without additional configuration. Special care should be taken to the interplay of the Intel, PGI, and GCC compilers, scientific libraries, and Python. The details for specific packages must be discussed with future maintainers. Some examples are shown below. **** TODO Intel Specific Environment Varaibales Set the following variavbles for Intel compilers and their dependencies: #+BEGIN_EXAMPLE CPATH GDBSERVER_MIC GDB_CROSS IDB_HOME INCLUDE INTEL_LICENSE_FILE IPPROOT LD_LIBRARY_PATH LIBRARY_PATH MANPATH MIC_LD_LIBRARY_PATH MIC_LIBRARY_PATH MKLROOT MKL_INCLUDE MKL_LIBRARY_PATH NLSPATH PATH TBBROOT VTUNE_AMPLIFIER_XE_2013_DIR #+END_EXAMPLE **** TODO MPICH2 Specific Environment Varaibales Are there specific environment variables? Or are the more generic family variables sufficient? **** IN-PROGRESS OpenMPI Specific Environment Varaibales According to [[[Open_MPI_wrappers]]], the following variables should be set #+BEGIN_EXAMPLE MPICC=mpicc MPICXX=mpic++ # MPICXX=mpiCC # MPICXX=mpicxx MPIF77=mpif77 MPIF90=mpif90 MPIFC=mpifort MPIFORTRAN=mpifort #+END_EXAMPLE The variables MPI_COMPILE_FLAGS and MPI_LINK_FLAGS should not be controlled by the environment modules. These variables can be assigned, if needed, by the users [[[Open_MPI_without-wrappers]]]: #+BEGIN_EXAMPLE MPI_COMPILE_FLAGS = $(shell mpicc --showme:compile) MPI_LINK_FLAGS = $(shell mpicc --showme:link) my_app: my_app.c $(CC) $(MPI_COMPILE_FLAGS) my_app.c $(MPI_LINK_FLAGS) -o my_app # Examples: mpicc --showme:compile -I/opt/mpi/openmpi-1.6.5-gcc-4.8.2-compat/include -pthread mpicc --showme:link -pthread -L/opt/mpi/openmpi-1.6.5-gcc-4.8.2-compat/lib -lmpi -ldl -lm -lnuma -Wl,--export-dynamic -lrt -lnsl -lutil -lm -ldl #+END_EXAMPLE **** TODO PGI Specific Environment Varaibales #+BEGIN_EXAMPLE LD_LIBRARY_PATH LM_LICENSE_FILE PATH MANPATH PGI PGI_INCLUDE PGI_LIB PGI_LIBS0 PGRSH #+END_EXAMPLE **** STOPPED System Specific Environment Variables CLOSED: [2014-04-29 Tue 16:13] #+BEGIN_EXAMPLE TMPDIR #+END_EXAMPLE If /tmp may be not large enough (like on Merlin4), set suitable TMPDIR, e.g.: #+BEGIN_EXAMPLE TMPDIR=/scratch/tmp # where /scratch ->/home/scratch #+END_EXAMPLE Note (Achim): TMPDIR *should not* be changed by an environment module. Valeri: agreed. ** IN-PROGRESS Must Support Modules at Multiple Locations (Local and Network Installations) *** IN-PROGRESS Must Support Combinations of Local and Network Installations It must be possible to install locally any selfconsistent subset of environment modules. The user should be able to select which installation is used if multiple installations are available. The should be a mechanism (e.g., environment variable) to control which installation is used on per-user basis. *** DONE Top Level Directory :PROVIDER_PREFIX: CLOSED: [2014-04-28 Mon 11:22] The top level directory is */opt/*, where *=psi*, in accordance with [[[FHS]]]. The top level directory may be a link to a local or network file system. See [[[TLD]]] for the PSI specific implementation details. *** IN-PROGRESS Must Support Modules in Home Directory User should be able to install their own modules. ** DONE May Support Runtime Environments when Complete Development Environments are not Desirable CLOSED: [2014-04-28 Mon 11:24] The end users may wish to run applications without loading all modules required to build these applications. To this end, run-time modules may be provided if the module maintainer decides so. ** DONE May Support HW Compartibility Requirements CLOSED: [2014-04-28 Mon 11:25] This functionality should allow one to hide environment modules that lack certain capabilities on a given HW platform. For example, all 64-bit applications should be hidden on a 32-bit system. A less trivial example is checking the CPU flags and hiding the applications compiled with options that are not supported by the CPU. If possible, make it "should support" requirement. ** IN-PROGRESS May Support Optional Optimization and Development Requirements There may be a mechanism to hide (default) or unhide modules, which are not intended for general use, e.g. of development and testing type, from the standard workflow. Note(Achim): Hiding should be implemented via MODULEPATH. Modules in testing state should print a warning. Everything else will end in a nightmare. Valeri: this paragraph should go to Implementation. ** DONE Must Define Minimum System Requirements to Support Generic Modules CLOSED: [2014-04-28 Mon 11:42] Example: CPU family XXX and SL6 or higher on 64-bit arch OR MacOS XYZ OR ... * IN-PROGRESS Design Proposal <> ** IN-PROGRESS Must Support Module Hierarchy *** IN-PROGRESS Must Support the Hierarchy Compiler-MPI-Application As defined in the reference implementations of the extended TCL modules [[[env_modules_psi]]] and Lmod [[[lmod]]]. *** IN-PROGRESS Must Support family Functionality As defined in the reference implementations of the extended TCL modules [[[env_modules_psi]]] and Lmod [[[lmod]]]. *** IN-PROGRESS May/Should Support Subcommand swap on Upper Levels of Module Hierarchy As defined in the reference implementations of Lmod [[[lmod]]]. ** IN-PROGRESS Molule Command Both the extended TCL modules and Lua modules (Lmod) can be supported. The user will have a choice of default module command and will be able to swith to the alternative if both implementations are available. *** IN-PROGRESS Extended TCL Modules See [[[env_modules_psi]]] and the reference implementations on Merlin4 (FIXME) and MacOS (FIXME). The extended TCL modules cannot not support Lua modules. *** IN-PROGRESS Lua Modules (Lmod) See [[[lmod]]] and the reference implementaion on Merlin4 (FIXME). The Lua modules may support (extended) TCL modules. ** IN-PROGRESS File System Layout *** IN-PROGRESS Top Level Directory :OPT_PROVIDER=/opt/psi: <> The top level directory for the PSI implememtation is */opt/psi* in accordance with [[[FHS]]]. : PROVIDER_PREFIX=/opt/psi The top level directory may be a link to a local file system or a link to AFS (GPFS). *** IN-PROGRESS Modulefile Root Directory. There must be *only one* modulefile root directory MODULEPATH_ROOT within a given implementation of environment modules: : MODULEPATH_ROOT=$OPT_PROVIDER/modulefiles For the PSI implementation *MODULEPATH_ROOT=/opt/psi/modulefiles*. Note (Achim): Actually I like the idea of something like a MODULEROOT_PATH. It should be possible to implement multiple hierarchies. This can be easly implemented with the extendet TCL Modules, I guess it wouldn't be to hard to implement this with Lua-Modules. Valeri: 'Your search - "MODULEROOT_PATH" - did not match any documents.' *** IN-PROGRESS The Layout of Subdirectories Used for Configuration of Environment Modules #+BEGIN_EXAMPLE /opt/psi OPT_PROVIDER :-- modulefiles MODULEPATH_ROOT : : : :-- Core ### Compilers and compiler-independent SW : : :-- gcc : : : :-- 4.8.2 : : : :-- x.y.z : : :-- intel : : : :-- x.y.z : : :-- pgi : : : :-- x.y.z : : : : : :-- foo : : : :-- x.y.z : : :-- bar : : : :-- x.y.z : : : :-- Compiler ### Compiler-independent SW : : :-- gcc : : : :-- 4.8.2 : : : :-- mpi : : : :-- openmpi-1.6.4 : : : :-- openmpi-1.8.0 : : :-- intel : : : :-- 14.0 : : : :-- mpi : : : :-- openmpi-1.6.4 : : : :-- openmpi-1.8.0 : : : :-- MPI ### Compiler- and MPI-independent SW : : :-- gcc : : : :-- 4.8.2 : : : :-- openmpi : : : :-- 1.6.4 : : : :-- scalasca : : : |-- 1.4.2 : : #+END_EXAMPLE *** IN-PROGRESS The Layout of Subdirectories Used for Software Installation Each version of each package should have its own installation directory, if possible (the exceptions will be discussed below), specified by the corresponding PREFIX. PREFIX should 1. be easy to understand by an expert user; 2. make it easier to write portable module scripts; 3. avoid unneeded layers of abstarction. Example: #+BEGIN_EXAMPLE /opt/psi OPT_PROVIDER : : ............................................................... PREFIX Subdirectories : : : :-- gcc : : :-- 4.8.2 ### PREFIX=$OPT_PROVIDER/$COMPILER/$COMPILER_VERSION : : :-- x.y.z : : : :-- intel : : :-- 14.0 : : : :-- openmpi : : :-- openmpi-1.6.4-gcc-4.8.2 ### PREFIX=$OPT_PROVIDER/$MPI/$MPI-$MPI_VERSION-$COMPILER-$COMPILER_VERSION : : :-- ... : : :-- openmpi-1.6.4-intel-14.0 : : :-- ... : : : :-- foo : : :-- foo-x.y.z-openmpi-1.6.4-intel-14.0 ### PREFIX=$OPT_PROVIDER/$NAME/$NAME-$VERSION-$MPI-$MPI_VERSION-$COMPILER-$COMPILER_VERSION : : : :-- bar : : :-- bar-x.y.z-intel-14.0 ### PREFIX=$OPT_PROVIDER/$NAME/$NAME-$VERSION-$COMPILER-$COMPILER_VERSION : : #+END_EXAMPLE **** IN-PROGRESS Recommended Layouts Mainterners may select any of the recommended installation layouts (defined by PREFIX). FIXME **** IN-PROGRESS Special Cases Some of the Intel tolls depend on services that must be properly configured and started by the init process. It means that only some subset of intel versions may provide full functionality of some tools (e.g. Vtune Amplifier). The environment modules will not handle this issue. ** STOPPED Environment Variables Reserved for Internal Use CLOSED: [2014-04-29 Tue 17:10] FIXME #+BEGIN_EXAMPLE PSI_COMPILER PSI_COMPILER_VERSION PSI_MPI PSI_MPI_VERSION #+END_EXAMPLE Note (Achim): Use case? Valeri: I use them in my Lua modules. The prefix PSI_ can be removed, then the variables defined in [[[Family]]] are sufficient. : PREFIX=$OPT_PROVIDER/$NAME/$NAME-$VERSION-$MPI-$MPI_VERSION-$COMPILER-$COMPILER_VERSION ** TODO Mainterners of Environment Modules and Their Responsibilities FIXME ** TODO Reference Implementations FIXME * TODO Implementation Proposal <<> FIXME ** IN-PROGRESS Must Provide Build Environment for All Supported Computing Environments ** IN-PROGRESS Build Environment Should Support Regression Tests Should be done by maintaners. ** DONE Build and Test Environment for Each Module Must Be Available for All Maintaners CLOSED: [2014-04-28 Mon 11:48] There must be no duplication of efforts. Every build must be reproducable. ** IN-PROGRESS Should Provide Facility for Phasing Out Obsolete and Buggy Modules * IN-PROGRESS References 1. <> http://modules.sourceforge.net/ 2. <> https://intranet.psi.ch/AIT/EnvironmentModules 3. <> https://www.tacc.utexas.edu/tacc-projects/lmod 4. <> http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard 5. <> http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html 6. <> http://www.open-mpi.org/faq/?category=mpi-apps 7. <> http://www.open-mpi.org/faq/?category=mpi-apps#cant-use-wrappers 8. <> https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpe2-1.0.7rc1/INSTALL