28 KiB
- Request for Comments: Environment Modules at PSI
- Introduction
- Specification
- Must Preserve the Core Functionality of Traditional Modulel
- Must Control the Availability of Modules Subject to Current State and Dependencies
- Settings for Standard and Application Specific Environment Variables
- Must Support Modules at Multiple Locations (Local and Network Installations)
- Shoud Support Runtime Environments without Going into Deatails of Development Environments
- Should Support HW Compartibility Requirements
- Must Define Minimum System Requirements to Support Generic Modules
- Should Support Optional Optimization and Development Requirements
- Design Proposal
- Implementation Proposal
- References
#+TITLE Project Environment Modules
Request for Comments: Environment Modules at PSI
Achim Gsell, Valeri Markushin, Hans Christian Stadler
Scientific Computing, AIT, PSI
2014-05-01
DONE Introduction
CLOSED: [2014-04-30 Wed 15:40]
This document describes the requirements for a new implementation of environment modules at PSI that should address known problems with the present implementation based on the classical TCL modules [env_modules_tcl]. A detailed overview of suggested solutions was presented in the talk [env_modules_psi] by Achim.
The details of the specification are described in Section [Specification]. The draft of design is presented in Section [Design].
There are two readily available candidates to meet the requrements of this project: the extended version of TCL modules [env_modules_psi] and the Lua based Lmod [lmod]. The implementation proposal is summarized in Section [Implementation]. The implementation detailes will be written when the specifications are fixed and a genelal agreement concerning the design is reached.
The tecnical details concerning building specific software packages are mostly beyond the scope of this document: here we are interested only in the minimum and complete set of environment variables and dependencies required to provide and use a particular package.
Our goal is to provide the specifications and design that make the job of maintaners easier and staightforward.
IN-PROGRESS Specification
<<Specification>>
Since we have a work in progress, we call it specification, not evaluation.
DONE Must Preserve the Core Functionality of Traditional Modulel
CLOSED: [2014-04-30 Wed 16:15]
The essential (to be specified) tradinional module subcommands must be supported. Both the traditional TCL and Lua modules meet this requirement. It is desirable to support all traditional subcommands in extensions of the existing implementations.
IN-PROGRESS Essential subcommands
add | load avail rm | del | unload help list show swap | switch use unuse whatis
Note that the swap subcommand is provided by both the raditional TCL modules and Lmod, but with different semantics.
DONE The TCL module subcommands
CLOSED: [2014-04-30 Wed 15:45]
module help Modules Release 3.2.10 2012-12-21 (Copyright GNU GPL v2 1991): Usage: module [ switches ] [ subcommand ] [subcommand-args ] Switches: -H|--help this usage info -V|--version modules version & configuration options -f|--force force active dependency resolution -t|--terse terse format avail and list format -l|--long long format avail and list format -h|--human readable format avail and list format -v|--verbose enable verbose messages -s|--silent disable verbose messages -c|--create create caches for avail and apropos -i|--icase case insensitive -u|--userlvl <lvl> set user level to (nov[ice],exp[ert],adv[anced]) Available SubCommands and Args: + add|load modulefile [modulefile ...] + rm|unload modulefile [modulefile ...] + switch|swap [modulefile1] modulefile2 + display|show modulefile [modulefile ...] + avail [modulefile [modulefile ...]] + use [-a|--append] dir [dir ...] + unuse dir [dir ...] + update + refresh + purge + list + clear + help [modulefile [modulefile ...]] + whatis [modulefile [modulefile ...]] + apropos|keyword string + initadd modulefile [modulefile ...] + initprepend modulefile [modulefile ...] + initrm modulefile [modulefile ...] + initswitch modulefile1 modulefile2 + initlist + initclear
DONE The Lmod module subcommands
CLOSED: [2014-04-30 Wed 15:57]
module help
Usage: module [options] sub-command [args ...]
Options:
-h -? -H --help This help message
--topic=topic help topics: modfuncs envvars
-D Program tracing written to stderr
--debug=dbglvl Program tracing written to stderr
-d --default List default modules only when used with avail
-q --quiet --expert Do not print out warnings
-t --terse Write out in machine readable format for commands: list, avail, spider, savelist
--initial_load loading Lmod for first time in a user shell
--latest Load latest (ignore default)
--ignore_cache Treat the cache file(s) as out-of-date
--novice Turn off expert flag
-w twidth --width=twidth Use this as max term width
-v --version Print version info and quit
-r --regexp use regular expression match
--dumpversion Dump version in a machine readable way and quit
--localvar=localvarA local variables needed to be set after this commands execution
--check_syntax --checkSyntax Checking module command syntax: do not load
--config Report Lmod Configuration
--mt Report Module Table State
--timer report run times
--force force removal of a sticky module or save an empty collection
module [options] sub-command [args ...]
Help sub-commands:
------------------
help prints this message
help module [...] print help message from module(s)
Loading/Unloading sub-commands:
-------------------------------
load | add module [...] load module(s)
try-load | try-add module [...] Add module(s), do not complain if not found
del | unload module [...] Remove module(s), do not complain if not found
swap | sw | switch m1 m2 unload m1 and load m2
purge unload all modules
refresh reload aliases from current list of modules.
update reload all currently loaded modules.
Listing / Searching sub-commands:
---------------------------------
list List loaded modules
list s1 s2 ... List loaded modules that match the pattern
avail | av List available modules
avail | av string List available modules that contain "string".
spider List all possible modules
spider module List all possible version of that module file
spider string List all module that contain the "string".
spider name/version Detailed information about that version of the module.
whatis module Print whatis information about module
keyword | key string Search all name and whatis that contain "string".
Searching with Lmod:
--------------------
All searching (spider, list, avail, keyword) support regular expressions:
spider -r '^p' Finds all the modules that start with `p' or `P'
spider -r mpi Finds all modules that have "mpi" in their name.
spider -r 'mpi$ Finds all modules that end with "mpi" in their name.
Handling a collection of modules:
--------------------------------
save | s Save the current list of modules to a user defined "default".
save | s name Save the current list of modules to "name" collection.
restore | r Restore modules from the user's "default" or system default.
restore | r name Restore modules from "name" collection.
restore system Restore module state to system defaults.
savelist List of saved collections.
Deprecated commands:
--------------------
reset The same as "restore system"
getdefault [name] load name collection of modules or user's "default" if no name given.
===> Use "restore" instead <====
setdefault [name] Save current list of modules to name if given, otherwise save as the default
list for you the user.
===> Use "save" instead. <====
Miscellaneous sub-commands:
---------------------------
show modulefile show the commands in the module file.
use [-a] path Prepend or Append path to MODULEPATH.
unuse path remove path from MODULEPATH.
tablelist output list of active modules as a lua table.
Important Environment Variables:
--------------------------------
LMOD_COLORIZE If defined to be "YES" then Lmod prints properties and warning in color.
---------------------------------------------------------------------------------------------------------------
The following guides are at http://www.tacc.utexas.edu/tacc-projects/lmod
User Guide - How to use.
Advance User Guide - How to create you own modules.
System Administrator Guide - How to install Lmod on your own system.
---------------------------------------------------------------------------------------------------------------
Modules based on Lua: Version 5.3.2 (5.3.2-2-ga7fbd80) 2014-03-21 22:04
by Robert McLay mclay@tacc.utexas.edu
DONE Must Control the Availability of Modules Subject to Current State and Dependencies
CLOSED: [2014-04-28 Mon 10:49]
The Module Hierarchy section has been moved to [DesignProposal].
DONE Must support SW Dependencies
CLOSED: [2014-04-28 Mon 10:46]
DONE Desirable to Support HW and OS Constaints
CLOSED: [2014-04-28 Mon 10:47]
DONE Desirable to Support Licensing Constaints
CLOSED: [2014-04-28 Mon 10:48]
Only nonfloating licenses will be taken into account.
IN-PROGRESS Settings for Standard and Application Specific Environment Variables
DONE Must Export Environment Variables Specifying Module Version and Prefix
CLOSED: [2014-04-30 Wed 16:42]
The following environment variables must be set by all modules, unless there are conflicts:
<NAME>_VERSION <NAME>_PREFIX <NAME>_DIR <NAME>_HOME
The last three variables are set to the same value that normally corresponds to PREFIX.
The conflicts must be resolved by the maintaners of the modules involved.
DONE Must Export Family Environment Variables
CLOSED: [2014-04-30 Wed 16:37] <<Family>>
Some environment modules can be grouped in nonoverlapping families. For each family, maximum one module can be loaded.
Families must have reserved names. The following list of families can be appended if needed:
COMPILER MPI
Each module belonging to a family must set the following environment variables:
<FAMILY>=<implementation> <FAMILY>_VERSION=<implementation-version> <FAMILY>_DIR=<implementation-prefix>
E.g. for modules providing compilers:
COMPILER COMPILER_VERSION COMPILER_DIR
and for modules providing MPI:
MPI MPI_VERSION MPI_DIR
Examples:
COMPILER=gcc COMPILER_VERSION=4.8.2 MPI=openmpi MPI_VERSION=1.8.0
WAITING Must Export a Predefined Subset of POSIX Standard and Other Common Environment Variables
DONE POSIX and POSIX-like
CLOSED: [2014-04-30 Wed 17:15]
Every module that has effect on the following POSIX standard environment variables (FIXME) must explicitly export them:
PATH LD_LIBRARY_PATH MANPATH
WAITING COMPILER Family
<<COMPILER_EM>>
See [Open_Group_Base_Specifications_EM] for the list of variables that are frequently exported by widely used command interpreters and applications. In particular, the following variables are considered as widely used:
CC CFLAGS FC FFLAGS LFLAGS MAKEFLAGS MANPATH PATH
The flags should not be set by the environment modules because the users, who need them, will know better how to set the flags. The compiler default options will use implicitly some flags and the modules should not intefere with this mechanism.
FIXME (non-POSIX, e.g. common GCC shared by many other applications) if corresponding path exists:
CPATH C_INCLUDE_PATH CPLUS_INCLUDE_PATH <NAME>_INCLUDE_DIR LIBRARY_PATH <NAME>_LIBRARY_DIR LD_LIBRARY_PATH
The difference between _DIR and _PATH: the former means a path to a single directory, the latter can be a column-separated list of paths.
Summary of the environment variables that must be exported for each module of the COMPILER family:
CC CXX F77 F90 FC FORTRAN CPATH C_INCLUDE_PATH CPLUS_INCLUDE_PATH <NAME>_INCLUDE_DIR LIBRARY_PATH <NAME>_LIBRARY_DIR LD_LIBRARY_PATH
DONE MPI Family
CLOSED: [2014-04-30 Wed 17:27] <<MPI_EM>>
The environment variables for MPI wrappers begin with MPI and must be set by each module of the MPI family:
MPICC MPICXX MPIF77 MPIF90 MPIFC MPIFORTRAN MPIEXEC MPIRUN
The environment variables for MPI wrappers can also begin with MPI_ [ANL_MPICH2]:
MPI_CC MPI_CXX MPI_F77 MPI_F90 MPI_FC MPI_FORTRAN MPI_EXEC MPI_RUN
The variables MPI_INC and MPI_LIBS are used, but should not be set by the environment modules because they can be easily set by the user if needed.
The names without underscore seem to be more common, but the underscore is offen used in MPICH2.
| Search term | "MPICC=mpicc" | "MPI_CC=mpicc" |
| Nunber of results | 4490 | 922 |
| Reference | [ANL_MPICH2] |
The names with MPI_ may be set if the maintaner decides to provide them (for a good reason).
The environment variables should allow to distiguish between MPI wrappers (e.g. MPICC) and ordinary compilers (e.g. CC).
STOPPED Should Export Module Identifiers
CLOSED: [2014-04-30 Wed 16:19] <<ModuleIdentifiers>>
This part was merged with [Family].
DONE Should Export Package- or Vendor-Defined Environment Variables
CLOSED: [2014-04-30 Wed 16:54]
Each module should set the environment variables that make its use easier for the user without additional configuration.
Special care should be taken to the interplay of the Intel, PGI, and GCC compilers, scientific libraries, and Python.
The details for specific packages must be discussed with future maintainers. Some examples are shown below.
DONE Intel Specific Environment Varaibales
CLOSED: [2014-04-30 Wed 16:59]
In addition to the above specified variables, that must be set for all compilers [COMPILER_EM], set the following variavbles for Intel compilers (and their dependencies):
CPATH GDBSERVER_MIC GDB_CROSS IDB_HOME INCLUDE INTEL_LICENSE_FILE IPPROOT LD_LIBRARY_PATH LIBRARY_PATH MANPATH MIC_LD_LIBRARY_PATH MIC_LIBRARY_PATH MKLROOT MKL_INCLUDE MKL_LIBRARY_PATH NLSPATH PATH TBBROOT VTUNE_AMPLIFIER_XE_2013_DIR
These variables are set by the Intel configuration scripts, e.g. compilervars.sh, amplxe-vars.sh, advixe-vars.sh, etc.
The list of Intel specific environment variables must be defined and supported by the maintaner(s) of Intel packages. Special care should be talen to avoid conflicts with other packages. Every conflict that may lead to a behaviour, that is different from the native Intel configuration, must be documented.
WAITING MPICH2 Specific Environment Varaibales
Are there specific environment variables? Or are the more generic family variables sufficient?
This issue can wait until this MPI implementation is requested.
DONE OpenMPI Specific Environment Varaibales
CLOSED: [2014-04-30 Wed 17:33]
The module name for OpenMPI modules is openmpi (low case), for example:
MPI=openmpi MPI_VERSION=1.8.0
The recommended rules for MPI_PREFIX are discussed below.
According to [MPI_EM] and [Open_MPI_wrappers], the following variables for MPI wrappers must be set by each OpenMPI module
MPICC=mpicc MPICXX=mpic++ # MPICXX=mpiCC # MPICXX=mpicxx MPIF77=mpif77 MPIF90=mpif90 MPIFC=mpifort MPIFORTRAN=mpifort
The variables MPI_COMPILE_FLAGS and MPI_LINK_FLAGS should not be controlled by the environment modules. These variables can be assigned, if needed, by the users [Open_MPI_without-wrappers]:
MPI_COMPILE_FLAGS = $(shell mpicc --showme:compile)
MPI_LINK_FLAGS = $(shell mpicc --showme:link)
my_app: my_app.c
$(CC) $(MPI_COMPILE_FLAGS) my_app.c $(MPI_LINK_FLAGS) -o my_app
# Examples:
mpicc --showme:compile
-I/opt/mpi/openmpi-1.6.5-gcc-4.8.2-compat/include -pthread
mpicc --showme:link
-pthread -L/opt/mpi/openmpi-1.6.5-gcc-4.8.2-compat/lib -lmpi -ldl -lm -lnuma -Wl,--export-dynamic -lrt -lnsl -lutil -lm -ldl
IN-PROGRESS PGI Specific Environment Varaibales
In addition to the configuration scripts, PGI provides its own modulefiles.
LD_LIBRARY_PATH LM_LICENSE_FILE PATH MANPATH PGI PGI_INCLUDE PGI_LIB PGI_LIBS0 PGRSH
The list of PGI specific environment varaibles must be defined by the maintaner.
STOPPED System Specific Environment Variables
CLOSED: [2014-04-29 Tue 16:13]
The environment modules must not set system specific variables, like TMPDIR.
If /tmp may be not large enough (like on Merlin4), set suitable TMPDIR on the corresponding systems, e.g.:
TMPDIR=/scratch/tmp # where /scratch ->/home/scratch
DONE Must Support Modules at Multiple Locations (Local and Network Installations)
CLOSED: [2014-05-01 Thu 10:46]
DONE Must Support Combinations of Local and Network Installations
CLOSED: [2014-05-01 Thu 10:41]
It must be possible to install locally any selfconsistent subset of environment modules.
The user should be able to select which installation is used if multiple installations are available.
There should be a mechanism (e.g., environment variable) to control which installation is used on per-user basis.
DONE Top Level Directory PROVIDER_PREFIX
DONE Must Support Modules in User Home Directory
CLOSED: [2014-05-01 Thu 10:42]
Users should be able to install their own modules.
DONE Shoud Support Runtime Environments without Going into Deatails of Development Environments
CLOSED: [2014-04-28 Mon 11:24]
The user should be able to load default modules in their runtime environments without going through the chain of dependent modules. For example, the user should be able to load the default version of a module that depends on compiler and MPI without explicitly loading the corresponding versions of the compiler and the MPI library.
DONE Should Support HW Compartibility Requirements
CLOSED: [2014-04-28 Mon 11:25]
This functionality should allow one to hide environment modules that lack certain capabilities on a given HW platform. For example, all 64-bit applications should be hidden on a 32-bit system. A less trivial example is checking the CPU flags and hiding the applications compiled with options that are not supported by the CPU.
IN-PROGRESS Must Define Minimum System Requirements to Support Generic Modules
The minimum HW requirements for 64-bit systems: Intel and AMD CPUs with the support for the GCC option -mtune=core2 that includes the 64-bit extensions, MMX, SSE, SSE2, SSE3 and SSSE3 instruction set support [man_gcc].
The minimum OS requirement: Linux SL6 64-bit or higher, MacOS …
Most modules built for SL6 are expected to work in SL5 and SL7, but the compatibility with SL5 may not be guaranteed.
Do we need to support 32-bit systems?
DONE Should Support Optional Optimization and Development Requirements
CLOSED: [2014-05-01 Thu 11:31]
The modules that are not intended for general use, e.g. of development and testing type, should be hidden unless the user explicitly decides to use them.
DONE Design Proposal
CLOSED: [2014-05-01 Thu 13:45] <<DesignProposal>>
DONE Must Support Module Hierarchy
CLOSED: [2014-05-01 Thu 13:45]
DONE Must Support family Functionality
CLOSED: [2014-05-01 Thu 13:30]
As defined in the reference implementations of the extended TCL modules [env_modules_psi] and Lmod [lmod].
DONE Must Support the Hierarchy Compiler-MPI-Application
CLOSED: [2014-05-01 Thu 12:27]
As defined in the reference implementations of the extended TCL modules [env_modules_psi] and Lmod [lmod].
DONE Should Support Subcommand swap on Upper Levels of Module Hierarchy
CLOSED: [2014-05-01 Thu 13:30]
As defined in the reference implementations of Lmod [lmod].
This requirement implies that the recommended implementation of environment modules should support the same swap functionality as Lmod.
DONE Molule Command
CLOSED: [2014-05-01 Thu 13:45]
DONE Only One Implementation of Environment Modules Should Be Officially Supported
CLOSED: [2014-05-01 Thu 13:44]
The user may be able to use alternative implementations at his own efforts. The maintaners may provide support for alternative implementations for their packages if they find it useful. There may be no official support for alternative implementations.
DONE Extended TCL Modules Should Be the Recommended Implementation
CLOSED: [2014-05-01 Thu 13:45]
The extended TCL modules is the recommended implementation.
See [env_modules_psi] and the reference implementations on Merlin4 (FIXME) and MacOS (FIXME).
The extended TCL modules cannot not support Lua modules.
DONE Lua Modules (Lmod)
CLOSED: [2014-05-01 Thu 13:45]
See [lmod] and the reference implementaion on Merlin4 (FIXME).
The Lua modules may support (extended) TCL modules.
The maintaners of particular packages may provide support for Lua if they find it useful.
IN-PROGRESS File System Layout
IN-PROGRESS Top Level Directory :PROVIDER_PREFIX=/opt/psi:
<<TLD>>
The top level directory for the PSI implememtation is /opt/psi in accordance with [FHS].
PROVIDER_PREFIX=/opt/psi
The top level directory may be a link to a local file system or a link to AFS (GPFS).
IN-PROGRESS Modulefile Root Directory.
There must be only one modulefile root directory MODULEPATH_ROOT within a given implementation of environment modules:
MODULEPATH_ROOT=$OPT_PROVIDER/modulefiles
For the PSI implementation MODULEPATH_ROOT=/opt/psi/modulefiles.
Note (Achim): Actually I like the idea of something like a MODULEROOT_PATH. It should be possible to implement multiple hierarchies. This can be easly implemented with the extendet TCL Modules, I guess it wouldn't be to hard to implement this with Lua-Modules. Valeri: 'Your search - "MODULEROOT_PATH" - did not match any documents.'
IN-PROGRESS The Layout of Subdirectories Used for Configuration of Environment Modules
/opt/psi OPT_PROVIDER :-- modulefiles MODULEPATH_ROOT : : : :-- Core ### Compilers and compiler-independent SW : : :-- gcc : : : :-- 4.8.2 : : : :-- x.y.z : : :-- intel : : : :-- x.y.z : : :-- pgi : : : :-- x.y.z : : : : : :-- foo : : : :-- x.y.z : : :-- bar : : : :-- x.y.z : : : :-- Compiler ### Compiler-independent SW : : :-- gcc : : : :-- 4.8.2 : : : :-- mpi : : : :-- openmpi-1.6.4 : : : :-- openmpi-1.8.0 : : :-- intel : : : :-- 14.0 : : : :-- mpi : : : :-- openmpi-1.6.4 : : : :-- openmpi-1.8.0 : : : :-- MPI ### Compiler- and MPI-independent SW : : :-- gcc : : : :-- 4.8.2 : : : :-- openmpi : : : :-- 1.6.4 : : : :-- scalasca : : : |-- 1.4.2 : :
IN-PROGRESS The Layout of Subdirectories Used for Software Installation
Each version of each package should have its own installation directory, if possible (the exceptions will be discussed below), specified by the corresponding PREFIX.
PREFIX should
- be easy to understand by an expert user;
- make it easier to write portable module scripts;
- avoid unneeded layers of abstarction.
Example:
/opt/psi OPT_PROVIDER : : ............................................................... PREFIX Subdirectories : : : :-- gcc : : :-- 4.8.2 ### PREFIX=$OPT_PROVIDER/$COMPILER/$COMPILER_VERSION : : :-- x.y.z : : : :-- intel : : :-- 14.0 : : : :-- openmpi : : :-- openmpi-1.6.4-gcc-4.8.2 ### PREFIX=$OPT_PROVIDER/$MPI/$MPI-$MPI_VERSION-$COMPILER-$COMPILER_VERSION : : :-- ... : : :-- openmpi-1.6.4-intel-14.0 : : :-- ... : : : :-- foo : : :-- foo-x.y.z-openmpi-1.6.4-intel-14.0 ### PREFIX=$OPT_PROVIDER/$NAME/$NAME-$VERSION-$MPI-$MPI_VERSION-$COMPILER-$COMPILER_VERSION : : : :-- bar : : :-- bar-x.y.z-intel-14.0 ### PREFIX=$OPT_PROVIDER/$NAME/$NAME-$VERSION-$COMPILER-$COMPILER_VERSION : :
IN-PROGRESS Recommended Layouts
Mainterners may select any of the recommended installation layouts (defined by PREFIX).
FIXME
IN-PROGRESS Special Cases
Some of the Intel tolls depend on services that must be properly configured and started by the init process. It means that only some subset of intel versions may provide full functionality of some tools (e.g. Vtune Amplifier). The environment modules will not handle this issue.
STOPPED Environment Variables Reserved for Internal Use
CLOSED: [2014-04-29 Tue 17:10]
FIXME
PSI_COMPILER PSI_COMPILER_VERSION PSI_MPI PSI_MPI_VERSION
Note (Achim): Use case? Valeri: I use them in my Lua modules. The prefix PSI_ can be removed, then the variables defined in [Family] are sufficient.
PREFIX=$OPT_PROVIDER/$NAME/$NAME-$VERSION-$MPI-$MPI_VERSION-$COMPILER-$COMPILER_VERSION
TODO Mainterners of Environment Modules and Their Responsibilities
FIXME
TODO Reference Implementations
FIXME
TODO Implementation Proposal
<<<Implementation>>
FIXME
IN-PROGRESS Hiding of Testing Modules
Note(Achim): Hiding should be implemented via MODULEPATH. Modules in testing state should print a warning. Everything else will end in a nightmare. Valeri: this paragraph should go to Implementation.
IN-PROGRESS Must Provide Build Environment for All Supported Computing Environments
IN-PROGRESS Build Environment Should Support Regression Tests
Should be done by maintaners.
DONE Build and Test Environment for Each Module Must Be Available for All Maintaners
CLOSED: [2014-04-28 Mon 11:48]
There must be no duplication of efforts. Every build must be reproducable.
IN-PROGRESS Should Provide Facility for Phasing Out Obsolete and Buggy Modules
IN-PROGRESS References
- <<env_modules_tkl>> http://modules.sourceforge.net/
- <<env_modules_psi>> https://intranet.psi.ch/AIT/EnvironmentModules
- <<lmod>> https://www.tacc.utexas.edu/tacc-projects/lmod
- <<FHS>> http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
- <<Open_Group_Base_Specifications_EM>> http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html
- <<Open_MPI_wrappers>> http://www.open-mpi.org/faq/?category=mpi-apps
- <<Open_MPI_without-wrappers>> http://www.open-mpi.org/faq/?category=mpi-apps#cant-use-wrappers
- <<ANL_MPICH2>> https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpe2-1.0.7rc1/INSTALL
- <<man_gcc>> "man gcc".