Merge branch '32-overlay-implementation' into 'master'

Resolve "overlay implementation"

Closes #32

See merge request Pmodules/src!2
This commit is contained in:
2022-10-27 13:09:40 +00:00
39 changed files with 4146 additions and 6071 deletions
+1
View File
@@ -11,4 +11,5 @@ Downloads
tmp
tclIndex
pbuild.log
libpmodules.bash
*~
+142
View File
@@ -1,5 +1,147 @@
# Changelog of Pmodules
## Version 1.1.10 (not yet tagged)
* **modulecmd**
* *User visible changes*
* New options for `module search`.
* With the option `--group` the search can be restricted to a
group.
* With the option `--newest` only the newest versions are
displayed.
* `find` as alias for the sub-command `search` added.
* Bugfix: the sub-commands `whatis` and `keyword|apropos` were
broken by design.
* Bugfix: after loading a `Pmodules` module, it was not shown with
`module list`.
* Bugfix in scanning the depth of groups.
* Bugfix: after `module purge` the environment variable
`PMODULES_HOME` was not defined an more.
* Bugfix: source the shell init file only if a `Pmodules` module
is loaded.
* Bugfix: unsetting aliases in modulefiles was not handled
properly in `module purge`
* *Internal changes and fixes*
* initialisation error for bash and zsh fixed
* **build-system**
* *User visible changes*
* `modbuild` is now defined as function like `module`. Therefor no
`Pmodules` module must be loaded to build a module with `modbuild`
* The system can now be defined in the module (YAML) configuration
file.
* Build dependencies can (and should) now be specified with
`build_requires` in the YAML configuration file.
* Bugfix: cleanup of modulefiles in overlays fixed. A module can be in
more than one overlay. These overlays must be specified in the
module configuration file.
* Bugfix: querying dependencies from YAML configuration file
fixed. Under some conditions the string 'null' was in the list
of dependencies.
* Bugfix: create group directory if it doesn't exist.
* Bugfix: create the module `$PREFIX` before processing the
install targets not before all targets. If `$PREFIX` is created
before processing any target and the build fails, `modbuild`
assumes that the module have been already built successfully.
* *Internal changes and fixes*
* code review/re-factoring
* `modbuild` is now using the Bash installed in `Pmodules` itself.
* test code with `set -o nounset`, several issues with this
setting fixed (not necessarily bugs).
* **other changes**
* The build script to bootstrap Pmodules itself doesn't use modbuild
any more to compile required software packages. With this change
we can remove some special cases from modbuild.
* The bootstrap script requires Bash 5.0 or newer now.
* Bugfix: in the `Pmodules` modulefile force the sourcing of the
shell init script while in mode `load` only.
## Version 1.1.9
* **modulecmd**
* *User visible changes*
* Overlay info added to output of sub-command `search`.
* Output of `module search --verbose` revised for better readability.
* *Internal changes and fixes*
* The shell`s init file is sourced, when Pmodules is loaded as module.
This is required if there are changes in the module function or too
define new shell functions.
* A bug in `libmodules.tcl:module-addgroup()` which crashed
`module load ...` has been fixed.
* In versions before 1.1.9 a colon at the beginning or end of `MODULEPATH`
crashed the module function. This has been fixed.
* **build-system**
* *User visible changes*
* The command `modbuild` is now defined as shell function analog to
the `module` command. The main reason to introduce this function
is due to the fact that Bash version 5 or newer is now required
by `modbuild`. The function `modbuild` load Bash 5.x as module
before calling the modbuild-script. If you want to use the script
directly, a Bash binary with version 5.x must be in PATH.
* If a build-script is in the current working directory,
`modbuild` can now be called without specifying the build-script.
* In case of an error in a build-step the build process did not
abort as it should. This has been fixed.
* The option `--overlay` can now be used
- to define an overlay if legacy variants files are used
- to override the overlay in a YAML variants file.
* The new keyword `with` has been introduced in YAML variants file
to specified hierarchical dependencies.
* The function `pbuild::supported_os` has been
removed. `pbuild::supported_systems` provides the same
functionality for legacy configuration files. In YAML module
configuration files `systems` have to be used.
* **Internal changes and fixes**
* bugfix in setting `PATH`
* requires bash 5 or later
## Version 1.1.8
* **modulecmd**
* *User visible changes*
* configuration in YAML files
* modulefiles and software must not
have a common root directory
* the installation root must be specified, it doesn`t default
to the base 'overlay' any more.
* zsh initialisation fixed.
* *Internal changes and fixes*
* std::upvar() replaced with reference variables in part of the
code.
* environment variable `PMODULES_ROOT` removed.
* unsetting aliases fixed.
* update to bash 5.1.16
* update to findutils 4.9 (macOS only)
* minor fixes
* **build-system**
* *User visible changes*
* YAML format for variants files
* *Internal changes and fixes*
* use lib `libpmodules.bash`
* bugfixes
* **modmanage**
* *User visible changes*
* none, support for overlays still missing
* *Internal changes and fixes*
* none
## Version 1.1.7
* **modulecmd**
* list of available overlays in subcommand `use` is now better readable
* **buid-system**
* overlay definition must now be in YAML format
* support for YAML formatted variant files (the legacy format
is still supported)
* build-system in 1.1.6 was still work in progress and broken
## Version 1.1.6
* **modulecmd**
* bugfix in searching/loading modules in a hierarchical
group
## Version 1.1.5
* first public version with the overlay feature
## Version 1.0.0rc11
* **modulecmd**
* *User visible changes*
+5 -8
View File
@@ -11,10 +11,6 @@ fi
############################################################################
# some sanity checks
#
if [[ ! -d ${PMODULES_ROOT} ]]; then
echo "Oops: ${PMODULES_ROOT}: Set as Pmodules root, but this is not a directory." 1>&2
return 1
fi
if [[ ! -d ${PMODULES_HOME} ]]; then
echo "Oops: ${PMODULES_HOME}: Set as Pmodules home, but this is not a directory." 1>&2
return 1
@@ -33,13 +29,9 @@ fi
unset MODULE_VERSION
unset MODULE_VERSION_STACK
unset MODULESHOME
unset PMODULES_ENV
declare -x PMODULES_MODULEFILES_DIR='modulefiles'
declare -x PMODULES_CONFIG_DIR='config'
declare -x PMODULES_DIR="${PMODULES_HOME}"
#############################################################################
# implement module comand as shell function
#
@@ -49,6 +41,11 @@ module() {
}
export -f module
modbuild(){
"${PMODULES_HOME}/bin/modbuild" "$@"
}
export -f modbuild
# Local Variables:
# mode: sh
# sh-basic-offset: 8
+119 -50
View File
@@ -4,6 +4,7 @@
# unload modules if parent removed
#
if {[info exists env(PMODULES_DEBUG)] && $env(PMODULES_DEBUG)} {
proc debug {msg} {
set level [expr [info level] -2]
@@ -24,13 +25,35 @@ debug "loading libmodules"
package require base64
set ::MODULEFILES_DIR "modulefiles"
set ::ol_replacing "r"
proc _pmodules_parse_pmodules_env { } {
debug "enter"
#
# In this library we need the value of some BASH variables
# defined in PMODULES_ENV. In this function we translate
# these variables definitions - created in BASH with
# 'typeset -p VAR' - to Tcl.
#
foreach line [split [base64::decode $::env(PMODULES_ENV)] "\n"] {
if { ![regexp -- {.* -[aAx]* (.*)=\((.*)\)} $line -> name value] } {
continue
}
switch $name {
Dir2OverlayMap {
array set ::Dir2OverlayMap [regsub -all {[]=[]} $value " "]
}
OverlayInfo {
array set ::OverlayInfo [regsub -all {[]=[]} $value " "]
}
UsedOverlays {
array set tmp [regsub -all {[]=[]} $value " "]
set ::UsedOverlays {}
set l [lsort [array names tmp]]
foreach k $l {
lappend ::UsedOverlays $tmp($k)
}
}
UsedGroups {
set ::UsedGroups $value
}
@@ -55,14 +78,32 @@ proc module-addgroup { group } {
set ::${group}_version $version
if { [module-info mode load] } {
prepend-path MODULEPATH [file join \
$::PmodulesRoot \
$group \
$::PmodulesModulfilesDir \
{*}$::variant]
set overlays_to_add {}
foreach overlay $::UsedOverlays {
lappend overlays_to_add $overlay
set ol_type $::OverlayInfo($overlay:type)
debug "ol_type=$ol_type"
if { [string compare $ol_type $::ol_replacing] == 0 } {
break
}
}
foreach overlay [lreverse_n $overlays_to_add 1] {
debug "overlay=$overlay"
debug "group=$group"
debug "::variant=$::variant"
set dir [file join \
$::OverlayInfo($overlay:mod_root) \
$group \
$::MODULEFILES_DIR \
{*}$::variant]
debug "dir=$dir"
if { [file isdirectory $dir] } {
debug "prepend $dir to MODULEPATH "
prepend-path MODULEPATH $dir
}
}
debug "end foreach"
prepend-path UsedGroups $group
debug "mode=load: new MODULEPATH=$env(MODULEPATH)"
debug "mode=load: new UsedGroups=$env(UsedGroups)"
} elseif { [module-info mode remove] } {
set GROUP [string toupper $group]
debug "mode=remove: hierarchical group '${GROUP}'"
@@ -83,14 +124,15 @@ proc module-addgroup { group } {
debug "mode=remove: no orphan modules to unload"
}
debug "mode=remove: $env(MODULEPATH)"
set dir [file join \
$::PmodulesRoot \
$group \
$::PmodulesModulfilesDir \
{*}$::variant]
debug "mode=remove: dir=$dir"
remove-path MODULEPATH $dir
debug "mode=remove: $env(UsedGroups)"
foreach overlay $::UsedOverlays {
set dir [file join \
$overlay \
$group \
$::MODULEFILES_DIR \
{*}$::variant]
debug "remove $dir"
remove-path MODULEPATH $dir
}
remove-path UsedGroups $group
debug "mode=remove: $env(UsedGroups)"
}
@@ -131,7 +173,7 @@ proc _pmodules_setenv { PREFIX name version } {
return
}
set NAME [string toupper $name]
set NAME [string toupper $name]
regsub -- "-" ${NAME} "_" NAME
if { ! [info exist ::dont-setenv] } {
@@ -154,6 +196,9 @@ proc _pmodules_setenv { PREFIX name version } {
if { [lsearch ${::dont-setenv} "${NAME}_HOME"] == -1 } {
setenv ${NAME}_HOME $PREFIX
}
if { [lsearch ${::dont-setenv} "${NAME}_ROOT"] == -1 } {
setenv ${NAME}_ROOT $PREFIX
}
} else {
debug "$PREFIX is not a directory"
}
@@ -274,13 +319,45 @@ proc ModulesHelp { } {
# intialize global vars
# Modulefile is something like
#
# ${PMODULES_ROOT}/group/${PMODULES_MODULEFILES_DIR}/name/version
# <root_dir>/group/modulefiles/name/version
# or
# ${PMODULES_ROOT}/group/${PMODULES_MODULEFILES_DIR}/X1/Y1/name/version
# <root_dir>/group/modulefiles/X1/Y1/name/version
# or
# ${PMODULES_ROOT}/group/${PMODULES_MODULEFILES_DIR}/X1/Y1//X2/Y2/name/version
# <root_dir>/group/modulefiles/X1/Y1//X2/Y2/name/version
#
proc _find_overlay { modulefile_components } {
debug "_find_overlay()"
foreach ol $::UsedOverlays {
debug "$ol"
set ol_mod_root $::OverlayInfo(${ol}:mod_root)
if { [string range $ol_mod_root end end] == "/" } {
set ol_mod_root [string range $ol_mod_root 0 end-1]
}
debug "$ol_mod_root"
set ol_mod_root_splitted [file split $ol_mod_root]
set modulefile_root [file join \
{*}[lrange \
$modulefile_components \
0 [expr [llength $ol_mod_root_splitted] - 1]]]
debug "$modulefile_root"
if { [string compare $ol_mod_root $modulefile_root] == 0 } {
debug "$ol_mod_root_splitted"
return $ol_mod_root_splitted
}
}
debug "not found"
return {}
}
proc _is_in_overlay { } {
debug "_is_in_overlay?"
set parts [_find_overlay [file split $::ModulesCurrentModulefile]]
debug "_is_in_overlay: $parts"
expr {[string compare $parts ""] == 0 }
}
proc _pmodules_init_global_vars { } {
debug "_pmodules_init_global_vars() called"
global group
global GROUP
global name
@@ -292,38 +369,31 @@ proc _pmodules_init_global_vars { } {
global V_PATCHLVL
global V_RELEASE
global V_PKG
global variant
global PREFIX # prefix of package
debug "$::ModulesCurrentModulefile"
set ::PmodulesRoot $::env(PMODULES_ROOT)
set ::PmodulesModulfilesDir $::env(PMODULES_MODULEFILES_DIR)
set modulefile [file split $::ModulesCurrentModulefile]
set pmodules_root [file split $::PmodulesRoot]
set pmodules_root_num_dirs [llength $pmodules_root]
set modulefile_splitted [file split $::ModulesCurrentModulefile]
set modulefile_root [file join {*}[lrange $modulefile 0 \
[expr $pmodules_root_num_dirs - 1]]]
if { $::PmodulesRoot != $modulefile_root } {
debug "stop sourcing: ${::PmodulesRoot} != $modulefile_root"
return
}
debug "modulefile is inside our root"
set rel_modulefile [lrange $modulefile [llength $pmodules_root] end]
set ol_mod_root_splitted [_find_overlay ${modulefile_splitted}]
set rel_modulefile [lrange $modulefile_splitted [llength $ol_mod_root_splitted] end]
set group [lindex $rel_modulefile 0]
set GROUP "${group}"
set name [lindex $modulefile end-1]
set name [lindex $modulefile_splitted end-1]
set P "${name}"
set version [lindex $modulefile end]
set version [lindex $modulefile_splitted end]
set V "${version}"
lassign [split $V -] V_PKG tmp
set V_RELEASE [lindex [split $tmp _] 0]
lassign [split $V_PKG .] V_MAJOR V_MINOR V_PATCHLVL
set variant [lrange $rel_modulefile 2 end]
set prefix "$pmodules_root $group [lreverse_n $variant 2]"
set variant [lrange $rel_modulefile 2 end]
set mod_root [file join {*}$ol_mod_root_splitted]
debug "mod_root=$mod_root"
set ol $::Dir2OverlayMap($mod_root)
debug "ol=$ol"
set install_prefix [file split $::OverlayInfo(${ol}:inst_root)]
set prefix "$install_prefix $group [lreverse_n $variant 2]"
set PREFIX [file join {*}$prefix]
debug "PREFIX=$PREFIX"
debug "group of module $name: $group"
}
@@ -331,14 +401,13 @@ if { [info exists ::whatis] } {
module-whatis "$whatis"
}
_pmodules_init_global_vars
#
# we cannot load another module with the same name
#
conflict $name
_pmodules_setenv ${PREFIX} ${name} ${version}
_pmodules_update_loaded_modules ${group} ${name} ${version}
_pmodules_parse_pmodules_env
if {[_is_in_overlay] == 0} {
debug "setup env vars for module in overlay"
_pmodules_init_global_vars
conflict $name
_pmodules_setenv ${PREFIX} ${name} ${version}
_pmodules_update_loaded_modules ${group} ${name} ${version}
}
debug "return from lib"
+1224 -1065
View File
File diff suppressed because it is too large Load Diff
-24
View File
@@ -1,24 +0,0 @@
#!/bin/bash
eval "pbuild::pre_prep_${system}() { :; }"
eval "pbuild::pre_prep_${OS}() { :; }"
eval "pbuild::post_prep_${system}() { :; }"
eval "pbuild::post_prep_${OS}() { :; }"
eval "pbuild::add_patch_${system}() { pbuild::add_patch \"\$@\"; }"
eval "pbuild::add_patch_${OD}() { pbuild::add_patch \"\$@\"; }"
eval "pbuild::pre_configure_${system}() { :; }"
eval "pbuild::pre_configure_${OS}() { :; }"
eval "pbuild::post_configure_${system}() { :; }"
eval "pbuild::post_configure_${OS}() { :; }"
eval "pbuild::pre_compile_${system}() { :; }"
eval "pbuild::pre_compile_${OS}() { :; }"
eval "pbuild::post_compile_${system}() { :; }"
eval "pbuild::post_compile_${OS}() { :; }"
eval "pbuild::pre_install_${system}() { :; }"
eval "pbuild::pre_install_${OS}() { :; }"
eval "pbuild::post_install_${system}() { :; }"
eval "pbuild::post_install_${OS}() { :; }"
+137 -14
View File
@@ -7,9 +7,19 @@ declare -A Subcommands=()
declare -A Options=()
declare -A Help=()
declare -a Overlays=()
declare -A OverlayInfo
declare -a UsedOverlays
declare -A Dir2OverlayMap
declare -r ol_normal='n'
declare -r ol_hiding='h'
declare -r ol_replacing='r'
# initialize help text of 'module --version'
Help['version']="
Pmodules @PMODULES_VERSION@ using Tcl Environment Modules @MODULES_VERSION@
Pmodules @PMODULES_VERSION@ using Tcl Environment Modules
Copyright GNU GPL v2
"
@@ -28,34 +38,147 @@ print_help() {
# $1: absolute path of a modulefile directory
#
compute_group_depth () {
local -r dir=$1
test -d "${dir}" || return 1
local -n result="$1"
local -r dir="$2"
if [[ ! -d "${dir}" ]]; then
${mkdir} -p "${dir}" || \
std::die 1 "Cannot create directory -- ${dir}"
fi
local group=${dir%/*}
local group=${group##*/}
[[ -n "${GroupDepths[${group}]}" ]] && return 0
local -i depth=$(${find} "${dir}" -depth \( -type f -o -type l \) \
result=$(${find} "${dir}" -depth \( -type f -o -type l \) \
-printf "%d" -quit 2>/dev/null)
(( depth-=2 ))
(( result-=2 ))
# if a group doesn't contain a modulefile, depth is negativ
# :FIXME: better solution?
(( depth < 0 )) && (( depth = 0 ))
GroupDepths[$group]=${depth}
(( result < 0 )) && (( result = 0 ))
}
#
# (Re-)Scan available groups in given root and compute group depth's
# (Re-)Scan available groups in given overlays and compute group depth's
#
# Args:
# $1: root of modulefile hierarchy
# $@: overlay names
#
scan_groups () {
local -r root="$1"
local moduledir
for moduledir in ${root}/*/${PMODULES_MODULEFILES_DIR}; do
compute_group_depth "${moduledir}"
local ol
local depth
for ol in "$@"; do
local mod_root="${OverlayInfo[${ol}:mod_root]}"
local dir
for dir in ${mod_root}/*/${PMODULES_MODULEFILES_DIR}; do
local group="${dir%/*}"
group="${group##*/}"
if [[ ! -v GroupDepths[${group}] ]]; then
compute_group_depth depth "${dir}"
GroupDepths[$group]=${depth}
fi
Dir2OverlayMap[${dir%/${PMODULES_MODULEFILES_DIR}*}]="${ol}"
done
done
}
pm::read_config(){
local -a config_files=()
_get_config_files(){
#
# return array with overlay configuration files
#
# Args:
# $1 [upvar] result
local -n fnames="$1"
# user defined via environment variable
if [[ -v PMODULES_OVERLAYS_DEF ]]; then
test -r "${PMODULES_OVERLAYS_DEF}" || \
std::die 3 \
"%s -- %s" \
"overlay definition file is not readable" \
"$_"
fnames+=("${PMODULES_OVERLAYS_DEF}")
fi
# user defined
if [[ -r "${HOME}/.Pmodules/Pmodules.yaml" ]]; then
fnames+=("${HOME}/.Pmodules/Pmodules.yaml")
fi
# system config file
test -r "${PMODULES_HOME%%/Tools*}/config/Pmodules.yaml" || \
std::die 3 \
"%s %s -- %s" \
"base overlay definition file" \
"does not exist or is not readable" \
"$_"
fnames+=("${PMODULES_HOME%%/Tools*}/config/Pmodules.yaml")
}
_get_ol_names(){
#
# get the names of all overlays
#
local -n fnames="$1"
${yq} -Ne eval-all '. as $item ireduce ({}; . *+ $item) |.Overlays|keys()' \
"${fnames[@]}" | awk '{print $2}'
}
_get_config_files config_files
eval $(std::parse_yaml "${config_files[-1]}" 'cfg_')
[[ -v cfg_DefaultGroups ]] && DefaultGroups="${cfg_DefaultGroups}"
[[ -v cfg_DefaultReleaseStages ]] && DefaultReleaseStages="${cfg_DefaultReleaseStages}"
[[ -v cfg_ReleaseStages ]] && ReleaseStages="${cfg_ReleaseStages}"
unset ${!cfg_*}
#
# loop over all overlays and save config in OverlayInfo and Dir2OverlayMap
#
# - at least install_root must be specified
# - modulefiles_root default so install_root
# - the type defaults to ${ol_normal}
#
Overlays=( $(_get_ol_names config_files) )
local ol=''
for ol in "${Overlays[@]}"; do
eval $(${yq} -Ne eval-all \
". as \$item ireduce ({}; . *+ \$item) |.Overlays.${ol}" \
"${config_files[@]}" | \
sed 's/: /=/; s/\(.*\)/local \1/')
[[ -n ${install_root} ]] || \
std::die 3 \
"install_root missing for overlay -- ${ol}"
[[ -d ${install_root} ]] || \
std::die 3 \
"Invalid installation root directory for overlay '${ol}' -- ${install_root}"
OverlayInfo[${ol}:inst_root]="${install_root}"
: ${modulefiles_root:=${install_root}}
[[ -d ${modulefiles_root} ]] || \
std::die 3 \
"Invalid modulefiles root directory for overlay '${ol}' -- ${modulefiles_root}"
OverlayInfo[${ol}:mod_root]="${modulefiles_root}"
Dir2OverlayMap[${modulefiles_root}]="${ol}"
: ${type:=${ol_normal}}
case ${type} in
${ol_normal} | ${ol_replacing} | ${ol_hiding} )
:
;;
* )
std::die 3 "Invalid type for overlay '${ol}' -- ${type}"
;;
esac
OverlayInfo[${ol}:type]="${type}"
OverlayInfo[${ol}:used]='no'
unset type modulefiles_root install_root
done
}
# Local Variables:
# mode: sh
# sh-basic-offset: 8
+79 -41
View File
@@ -20,7 +20,7 @@ std::error() {
}
std::debug() {
[[ ${PMODULES_DEBUG} ]] || return 0
[[ -v PMODULES_DEBUG ]] || return 0
std::log 2 "$@"
}
@@ -39,7 +39,7 @@ std::def_cmds(){
local path="$1"
shift
for cmd in "$@"; do
eval declare -g ${cmd}=$(PATH="${path}" which $cmd 2>/dev/null)
eval declare -gr ${cmd}=$(PATH="${path}" /usr/bin/which $cmd 2>/dev/null)
if [[ -z "${!cmd}" ]]; then
std::die 255 "${cmd} not found"
fi
@@ -78,42 +78,54 @@ std::get_abspath() {
}
std::append_path () {
local -r P=$1
local -r d=$2
local -nr P="$1"
shift 1
local dir
local dirs=''
for dir in "$@"; do
[[ "${P}" == @(|*:)${dir}@(|:*) ]] && continue
dirs+=":${dir}"
done
if ! echo ${!P} | egrep -q "(^|:)${d}($|:)" ; then
if [[ -z ${!P} ]]; then
export "$P=${d}"
else
export "$P=${!P}:${d}"
fi
fi
if [[ -z ${P} ]]; then
P="${dirs:1}" # remove leading ':'
else
P="${P}${dirs}"
fi
}
std::prepend_path () {
local -r P=$1
local -r d=$2
local -nr P="$1"
shift 1
if ! echo ${!P} | egrep -q "(^|:)${d}($|:)" ; then
if [[ -z ${!P} ]]; then
export "$P=${d}"
else
export "$P=${d}:${!P}"
fi
local dir
local dirs=''
for dir in "$@"; do
[[ "${P}" == @(|*:)${dir}@(|:*) ]] && continue
dirs+="${dir}:"
done
if [[ -z ${P} ]]; then
P="${dirs:0:-1}" # remove trailing ':'
else
P="${dirs}${P}"
fi
}
std::remove_path() {
local -r P=$1
local -r d=$2
local -nr P="$1"
shift 1
local -ar dirs="$@"
local new_path=''
local -r _P=( ${!P//:/ } )
# loop over all entries in path
for entry in "${_P[@]}"; do
[[ "${entry}" != "${d}" ]] && new_path+=":${entry}"
local dir=''
for dir in "${dirs[@]}"; do
# loop over all entries in path
for entry in "${_P[@]}"; do
[[ "${entry}" != "${dir}" ]] && new_path+=":${entry}"
done
done
# remove leading ':'
eval ${P}="${new_path:1}"
P="${new_path:1}" # remove leading ':'
}
#
@@ -163,22 +175,23 @@ std::replace_path () {
# analog to std::split_abspath() with a relative path.
#
std::split_path() {
local parts="$1"
local -r path="$2"
local -n parts="$1"
local -r path="$2"
IFS='/'
local std__split_path_result=( ${std__split_path_tmp} )
unset IFS
std::upvar ${parts} "${std__split_path_result[@]}"
parts="${std__split_path_result[@]}"
if (( $# >= 3 )); then
# return number of parts
std::upvar "$3" ${#std__split_path_result[@]}
local -n num="$3"
num="${#std__split_path_result[@]}"
fi
}
std::split_abspath() {
local parts="$1"
local -r path="$2"
local -n parts="$1"
local -r path="$2"
if [[ "${path:0:1}" == '/' ]]; then
local -r std__split_path_tmp="${path:1}"
else
@@ -188,16 +201,17 @@ std::split_abspath() {
IFS='/'
local std__split_path_result=( ${std__split_path_tmp} )
unset IFS
std::upvar ${parts} "${std__split_path_result[@]}"
parts="${std__split_path_result[@]}"
if (( $# >= 3 )); then
# return number of parts
std::upvar "$3" ${#std__split_path_result[@]}
local -n num="$3"
num="${#std__split_path_result[@]}"
fi
}
std::split_relpath() {
local parts="$1"
local -r path="$2"
local -n parts="$1"
local -r path="$2"
if [[ "${path:0:1}" == '/' ]]; then
std::die 255 "Oops: Internal error in '${FUNCNAME[0]}' called by '${FUNCNAME[1]}' }"
else
@@ -207,10 +221,11 @@ std::split_relpath() {
IFS='/'
local std__split_path_result=( ${std__split_path_tmp} )
unset IFS
std::upvar ${parts} "${std__split_path_result[@]}"
parts="${std__split_path_result[@]}"
if (( $# >= 3 )); then
# return number of parts
std::upvar "$3" ${#std__split_path_result[@]}
local -n num="$3"
num="${#std__split_path_result[@]}"
fi
}
@@ -274,11 +289,11 @@ std::upvar() {
}
std.get_os_release_linux() {
local lsb_release=$(which lsb_release)
#local lsb_release=$(which lsb_release)
local ID=''
local VERSION_ID=''
if [[ -n $(which lsb_release) ]]; then
if [[ -n $(which lsb_release 2>/dev/null) ]]; then
ID=$(lsb_release -is)
VERSION_ID=$(lsb_release -rs)
elif [[ -r '/etc/os-release' ]]; then
@@ -309,7 +324,7 @@ std::get_os_release() {
local -A func_map;
func_map['Linux']=std.get_os_release_linux
func_map['Darwin']=std.get_os_release_macos
${func_map[${OS}]}
${func_map[$(uname -s)]}
}
std::get_type() {
@@ -339,6 +354,29 @@ std::get_type() {
esac
}
std::parse_yaml() {
#
# parse a YAML file
# See: https://gist.github.com/pkuczynski/8665367
#
local -r fname="$1"
local -r prefix="$2"
local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
sed -ne "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" \
-e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p" "${fname}" |
awk -F$fs '{
indent = length($1)/2;
vname[indent] = $2;
for (i in vname) {
if (i > indent) {delete vname[i]}
}
if (length($3) > 0) {
vn="";
for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
}
}'
}
# Local Variables:
# mode: sh
# sh-basic-offset: 8
+390 -146
View File
@@ -1,4 +1,4 @@
#!/usr/bin/env bash
#!@BASH@
#
# The following build specific variables are set and used in libpbuild.bash:
# ARGS
@@ -7,38 +7,59 @@
#
#.............................................................................
set -x
declare VERSION='@PMODULES_VERSION@'
# get absolute path of script
declare mydir=$(dirname "$0")
declare -r mydir=$(cd ${mydir} && pwd -P)
# initialize PATH,
# add library installation directories to the PATH,
# so 'source' is able find them
if [[ $(uname -s) == 'Darwin' ]]; then
PATH='/usr/local/bin:'
else
PATH=''
fi
PATH+='/usr/bin:/bin:/usr/sbin:/sbin'
PATH+=":${mydir}"
PATH+=":${mydir}/../lib:${mydir}/../config"
source libstd.bash || {
source "${mydir}/../lib/libstd.bash" || {
echo "Oops: cannot source library -- '$_'" 1>&2; exit 3;
}
# can be set in the configuration file
declare PMODULES_DISTFILESDIR=''
declare PMODULES_TMPDIR=''
##############################################################################
#
# check availability of used commands and set environment variables
# cmd=$(which cmd)
# in the following we use these environment variable to call binaries.
#
declare -r MODULECMD="${PMODULES_HOME}/bin/modulecmd"
[[ -x ${MODULECMD} ]] || \
std::die 1 "Oops: modulecmd binary not available!"
source libpbuild.bash || \
std::def_cmds "${mydir}/../libexec" \
'yq'
std::def_cmds '/usr/bin:/bin:/usr/sbin:/sbin' \
'awk' 'base64' 'cat' 'cp' 'find' 'getopt' 'grep' \
'install' 'logger' 'make' 'mkdir' 'mktemp' 'patch' 'pwd' \
'rm' 'rmdir' 'sort' 'tar' 'tee' 'uname'
declare -r OS="$(${uname} -s)"
if [[ ${OS} == 'Darwin' ]]; then
std::def_cmds '/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin' \
'curl' 'otool' 'shasum' 'sysctl'
declare -r sha256sum="${shasum -a 256}"
else
std::def_cmds '/usr/bin:/bin:/usr/sbin:/sbin' \
'ldd' 'curl' 'sha256sum'
fi
# for the time being, we still set PATH. Just in case we forgot a binary
PATH='/usr/bin:/bin:/usr/sbin:/sbin'
##############################################################################
source "${mydir}/../lib/libpbuild.bash" || \
std::die 3 "Oops: cannot source library -- '$_'"
source "${mydir}/../lib/libpmodules.bash" || \
std::die 3 "Oops: cannot source library -- '$_'"
# save arguments, (still) required for building dependencies
declare -r ARGS="$@"
##############################################################################
set -o nounset
shopt -s nocaseglob
shopt -s extglob
shopt -s nullglob
##############################################################################
#
@@ -80,7 +101,7 @@ BUILD-STEPS OPTIONS:
--all
Run throu all steps including cleanup.
--update-modulefiles
-update-modulefiles
Only install the modulefile and set the release.
MISCELLANEOUS OPTIONS:
@@ -94,6 +115,9 @@ MISCELLANEOUS OPTIONS:
-v | --verbose )
Verbose output.
--debug )
Run in debug mode.
-j N | --jobs=N
Run N parallel make jobs.
@@ -119,11 +143,13 @@ MISCELLANEOUS OPTIONS:
remove all files created during building.
--distdir
Directory wwhere to store and lookup downloaded files.
Directory where to store and lookup downloaded files.
--tmpdir
Directory used for building a module.
--overlay
Install in this overlay. Defaults to '${PMODULES_HOME%%/Tools*}'.
"
exit 1
}
@@ -136,10 +162,13 @@ MISCELLANEOUS OPTIONS:
# then configuration file
# last default
# save arguments, required for building dependencies
declare -r ARGS="$@"
declare PMODULES_DISTFILESDIR=''
declare PMODULES_TMPDIR=''
# versions to be build, '.*' or none means all
declare -a versions=()
declare opt_bootstrap='no'
declare opt_build_config='modbuild.conf'
declare opt_build_target='all'
declare opt_dry_run='no'
declare opt_enable_cleanup_build='yes'
@@ -148,9 +177,13 @@ declare opt_force_rebuild='no'
declare -i opt_jobs=0
declare opt_update_modulefiles='no'
declare opt_system=''
declare opt_overlay=''
declare opt_verbose='no'
# array collecting all modules specified on the command line via '--with=module'
declare -a opt_with_modules=()
declare -- opt_config_file=''
declare BUILD_SCRIPT=''
parse_args() {
while (( $# > 0 )); do
@@ -181,13 +214,6 @@ parse_args() {
--dry-run )
opt_dry_run='yes'
;;
--config )
opt_build_config="$2"
shift 1
;;
--config=* )
opt_build_config="${1#*=}"
;;
--enable-cleanup )
opt_enable_cleanup_build='yes'
opt_enable_cleanup_src='yes'
@@ -208,51 +234,68 @@ parse_args() {
--disable-cleanup-src )
opt_enable_cleanup_src='no'
;;
--distdir )
PMODULES_DISTFILESDIR="$2"
shift
--distdir | --distdir=* )
if [[ $1 == *=* ]]; then
PMODULES_DISTFILESDIR="${1/--distdir=}"
else
PMODULES_DISTFILESDIR="$2"
shift
fi
;;
--distdir=* )
PMODULES_DISTFILESDIR="${1/--distdir=}"
--tmpdir | --tmpdir=* )
if [[ $1 == *=* ]]; then
PMODULES_TMPDIR="${1#--*=}"
else
PMODULES_TMPDIR="$2"
shift
fi
;;
--tmpdir )
PMODULES_TMPDIR="$2"
shift
--system | --system=* )
if [[ $1 == *=* ]]; then
opt_system="${1#--*=}"
else
opt_system="$2"
shift
fi
;;
--tmpdir=* )
PMODULES_TMPDIR="${1/--tmpdir=}"
--overlay | --overlay=* )
if [[ $1 == *=* ]]; then
opt_overlay="${1#--*=}"
else
opt_overlay="$2"
shift
fi
;;
--system )
opt_system="$2"
shift
--use-flags | --use-flags=* )
if [[ $1 == *=* ]]; then
USE_FLAGS=":${1#--*=}:"
else
USE_FLAGS=":$2:"
shift
fi
;;
--system=* )
opt_system="${1/*=}"
;;
--use-flags )
USE_FLAGS="y:$2:"
shift
;;
--use-flags=* )
USE_FLAGS=":${1/--use-flags=}:"
;;
--with )
opt_with_modules+=( "$2" )
shift
;;
--with=*/* )
m="${1/--with=}"
opt_with_modules+=( ${m} )
--with | --with=*/* )
if [[ $1 == *=* ]]; then
opt_with_modules+=( "${1#--*=}" )
else
opt_with_modules+=( "$2" )
shift
fi
;;
--prep | --configure | --compile | --install | --all )
opt_build_target=${1:2}
;;
--bootstrap )
opt_bootstrap='yes'
;;
--update-modulefiles )
opt_update_modulefiles='yes'
;;
--config-file | --config-file=* )
if [[ $1 == *=* ]]; then
opt_config_file=( "${1#--*=}" )
else
opt_config_file=( "$2" )
shift
fi
;;
-- )
:
;;
@@ -280,57 +323,72 @@ parse_args() {
esac
shift
done
[[ -n ${BUILD_SCRIPT} ]] || std::die 1 "No build-block specified!"
if [[ -z ${BUILD_SCRIPT} ]]; then
if [[ -r "${PWD}/build" ]]; then
if grep -q '#!.* modbuild' "${PWD}/build"; then
BUILD_SCRIPT="${PWD}/build"
BUILDBLOCK_DIR=$(dirname "${BUILD_SCRIPT}")
fi
fi
if [[ -z ${BUILD_SCRIPT} ]]; then
std::die 1 "Don't know what to build!"
fi
fi
(( ${#versions[@]} > 0)) || versions+=( '.*' )
}
shopt -s nocaseglob
find_variants_files(){
shopt -q nullglob || :
local -i nullglob_set=$?
shopt -s nullglob
local files=( "${BUILDBLOCK_DIR}"/*/"${BNAME_VARIANTS}"\.${opt_system} )
files+=( "${BUILDBLOCK_DIR}"/*/"${BNAME_VARIANTS}.$(uname -s)" )
local f
for f in "${BUILDBLOCK_DIR}"/*/"${BNAME_VARIANTS}"; do
[[ -e "${f}.${opt_system}" ]] \
|| [[ -e "${f}.$(uname -s)" ]] \
|| files+=( "$f" )
done
(( nullglob_set == 1 )) && shopt -u nullglob
std::upvar "$1" "${files[@]}"
#
# bash brace expansion of given args. Input args like:
#
# "text" "gcc/{9.3.0,10.3.0}" "openmpi/{4.0.5,4.1.0}"
#
# will be expanded to the following four lines:
#
# "text gcc/9.3.0 openmpi/4.0.5"
# "text gcc/9.3.0 openmpi/4.1.0"
# "text gcc/10.3.0 openmpi/4.0.5"
# "text gcc/10.3.0 openmpi/4.1.0"
#
bash_expand(){
local text="$1"
shift
local to_expand=( "${@}" )
if (( ${#to_expand[@]} == 0 )); then
echo ${text}
else
local list
eval list=( ${to_expand[0]} )
local s
for s in ${list[*]}; do
bash_expand "${text} ${s}" "${to_expand[@]:1}"
done;
fi;
}
shopt -s extglob
expand_variants_file(){
expand_deps(){
local text="$1"
shift
local deps=( "${@}" )
echo "${deps[@]}"
if (( ${#deps[@]} == 0 )); then
echo ${text}
else
local list
eval list=( ${deps[0]} )
for dep in ${list[*]}; do
expand_deps "${text} ${dep}" "${deps[@]:1}"
done
fi
build_modules_legacy() {
find_variants_files(){
local files=( "${BUILDBLOCK_DIR}"/*/"${BNAME_VARIANTS}"\.${opt_system} )
files+=( "${BUILDBLOCK_DIR}"/*/"${BNAME_VARIANTS}.$(uname -s)" )
local f
for f in "${BUILDBLOCK_DIR}"/*/"${BNAME_VARIANTS}"; do
[[ -e "${f}.${opt_system}" ]] \
|| [[ -e "${f}.$(uname -s)" ]] \
|| files+=( "$f" )
done
std::upvar "$1" "${files[@]}"
}
local -r input="$1"
while read -a toks; do
# skip empty and comment lines
[[ -z ${toks} ]] && continue
[[ ${toke:0:1} == '#' ]] && continue
# ignore first (module/version) and second (release) token
deps=( ${toks[*]:2} )
expand_deps "${toks[0]} ${toks[1]}" "${deps[@]}"
done < "${input}"
}
expand_variants_file(){
local -r input="$1"
while read -a toks; do
# skip empty and comment lines
[[ -z ${toks} ]] && continue
[[ ${toke:0:1} == '#' ]] && continue
local -a deps=( ${toks[*]:2} )
bash_expand "${toks[0]} ${toks[1]}" "${deps[@]}"
done < "${input}"
}
build_modules() {
local name="$1"
local version="$2"
local exact_match='no'
@@ -350,25 +408,17 @@ build_modules() {
local m
local pattern="/^${name}\/${version}[[:blank:]]/"
for m in "${with_modules[@]}"; do
if [[ -n $(awk "/${m%/*}[\/ ]/" "${files[@]}") ]]; then
if [[ -n $(${awk} "/${m%/*}[\/ ]/" "${files[@]}") ]]; then
pattern+=" && /${m//\//\\/}/"
fi
done
local variants=()
local variants_files=()
for f in "${files[@]}"; do
local line=''
while read line; do
variants+=( "${line}" )
variants_files+=( "$f" )
done < <(expand_variants_file "${f}" | awk "${pattern}")
# here we should add a check, whether the version of the
# found variants are in the right variants files. Example:
# a variant for hdf5/1.10.4 is not allowed in a variants file
# for version 1.8. For this we need the mapping of a variant
# to the variants file in array 'variants_files'
done < <(expand_variants_file "${f}" | ${awk} "${pattern}")
done
if (( ${#variants[@]} == 0 )); then
std::info "%s " \
@@ -383,6 +433,10 @@ build_modules() {
done
std::die 10 "Aborting..."
fi
declare ol_name='base'
declare ol_type='n'
declare ol_mod_root="${PMODULES_HOME%%/Tools*}"
declare ol_inst_root="${PMODULES_HOME%%/Tools*}"
local -i i=0
local -i num_variants=${#variants[@]}
for ((i = 0; i < num_variants; i++)); do
@@ -391,20 +445,224 @@ build_modules() {
version="${tokens[0]#*/}"
release="${tokens[1]}"
with_modules=( "${tokens[@]:2}" )
pbuild.build_module \
pbuild.build_module_legacy \
"${name}" "${version}" \
"${release}" "${with_modules[@]}"
done
}
build_modules_yaml(){
local -A mod_overlays=()
local -- fname="${opt_config_file:-${BUILDBLOCK_DIR}/files/config.yaml}"
[[ -r "${fname}" ]] || \
std::die 3 "YAML configuration file is not readable -- ${fname}"
echo "Using ${fname}..."
yaml_get_versions(){
local -n _result="$1"
local fname="$2"
local version="$3"
_result=( $(${yq} -Ne e \
"with_entries(select(.key | test(\"^${version}\$\")))|keys" \
"${fname}" 2>/dev/null | ${awk} '{print $2}') )
}
yaml_get_num_variants(){
local -n _result="$1"
local fname="$2"
local version="$3"
_result=$(${yq} -Ne e ".\"${version}\"|length" \
"${fname}" 2>/dev/null)
if (( $? != 0 )); then
_result=0
fi
}
yaml_get_relstage(){
local -n _result="$1"
local fname="$2"
local version="$3"
local idx="$4"
_result=$(${yq} -Ne e ".\"${version}\"[${idx}].relstage" \
"${fname}" 2>/dev/null)
(( $? == 0 )) && return
_result=$(${yq} -Ne e ".relstage" "${fname}" 2>/dev/null)
(( $? == 0 )) && return
_result='unstable'
}
yaml_get_overlay(){
local -n _result="$1"
local fname="$2"
local version="$3"
local idx="$4"
_result=$(${yq} -Ne e ".\"${version}\"[${idx}].overlay" \
"${fname}" 2>/dev/null)
(( $? == 0 )) && return
_result=$(${yq} -Ne e ".overlay" "${fname}" 2>/dev/null)
(( $? == 0 )) && return
_result='base'
}
yaml_get_group(){
local -n _result="$1"
local fname="$2"
local version="$3"
local idx="$4"
_result=$(${yq} -Ne e ".\"${version}\"[${idx}].group" \
"${fname}" 2>/dev/null)
(( $? == 0 )) && return
_result=$(${yq} -Ne e ".group" "${fname}" 2>/dev/null)
(( $? == 0 )) && return
_result=''
}
yaml_get_systems(){
local -n _result="$1"
local fname="$2"
local version="$3"
local idx="$4"
_result=( $(${yq} -Ne e ".\"${version}\"[${idx}].systems" \
"${fname}" 2>/dev/null) )
(( $? == 0 )) && return
_result=( $(${yq} -Ne e ".systems" "${fname}" 2>/dev/null) )
(( $? == 0 )) && return
_result=()
}
yaml_get_compilers(){
local -n _result="$1"
local fname="$2"
local version="$3"
local idx="$4"
_result=( $(${yq} -Ne e ".\"${version}\"[${idx}].compilers" \
"${fname}" 2>/dev/null) )
(( $? == 0 )) && return
_result=( $(${yq} -Ne e ".compilers" "${fname}" 2>/dev/null) )
(( $? == 0 )) && return
_result=()
}
yaml_get_dependencies(){
local -n _result="$1"
local fname="$2"
local version="$3"
local idx="$4"
_result=( $(${yq} -Ne e ".\"${version}\"[${idx}]|(.with, .dependencies)" \
"${fname}" 2>/dev/null) )
if (( $? != 0 )); then
# neither .with nor .dependencies are set
_result=()
return
fi
# if one of .with, .dependencies is not set, the vaulue is
# returned as 'null'.
local -i i
for i in "${!_result[@]}"; do
[[ ${_result[$i]} == 'null' ]] && unset -v '_result[$i]'
done
_result=("${_result[@]}")
}
yaml_get_build_requirements(){
local -n _result="$1"
local fname="$2"
local version="$3"
local idx="$4"
_result=( $(${yq} -Ne e ".\"${version}\"[${idx}].build_requires" \
"${fname}" 2>/dev/null) )
if (( $? != 0 )); then
_result=()
return
fi
}
local name="$1"
local version="$2"
shift 2
local with_modules=( $* )
local m
local pattern="//"
for m in "${with_modules[@]}"; do
pattern+=" && /${m//\//\\/}/"
done
local -a versions
yaml_get_versions versions "${fname}" "${name}/${version}"
for v in "${versions[@]}"; do
local -i n_variants
yaml_get_num_variants n_variants "${fname}" "${v}"
(( n_variants == 0 )) && (( n_variants = 1 ))
local -i i
local -a deps=()
local -a build_requires=()
local relstage
local group
local ol_name
for (( i=0; i<n_variants; i++)); do
if [[ -z ${opt_overlay} ]]; then
yaml_get_overlay ol_name "${fname}" "${v}" $i
else
ol_name="${opt_overlay}"
fi
if [[ ! -v OverlayInfo[${ol_name}:inst_root] ]]; then
std::die 3 "Overlay is not defined -- ${ol_name}"
fi
mod_overlays[$i]="${ol_name}"
done
for (( i=0; i<n_variants; i++)); do
ol_name="${mod_overlays[$i]}"
yaml_get_relstage relstage "${fname}" "${v}" $i
yaml_get_dependencies deps "${fname}" "${v}" $i
yaml_get_build_requirements build_requires "${fname}" "${v}" $i
yaml_get_group group "${fname}" "${v}" $i
pbuild.add_to_group "${group}"
# :FIXME:
# for the time being we prefix the build requirements
# with 'b:' and append them to the other dependencies.
# With this solution we don't have to change anything
# else in the code but have better readable variant
# files.
deps+=("${build_requires[@]/#/b:}")
declare ol_inst_root="${OverlayInfo[${ol_name}:inst_root]}"
declare ol_mod_root="${OverlayInfo[${ol_name}:mod_root]}"
local -a systems=()
local -a compilers=()
yaml_get_systems systems "${fname}" "${v}" $i
yaml_get_compilers compilers "${fname}" "${v}" $i
pbuild.supported_systems "${systems[@]}"
pbuild.supported_compilers "${compilers[@]}"
if (( ${#deps[@]} > 0 )); then
while read -a with_modules; do
pbuild.build_module_yaml \
"${name}" "${v##*/}" \
"${relstage}" "${with_modules[@]}"
done < <(bash_expand "" ${deps[@]}|${awk} "${pattern}")
else
pbuild.build_module_yaml \
"${name}" "${v##*/}" \
"${relstage}"
fi
done
done
}
build_modules() {
if [[ -n $(ls "${BUILDBLOCK_DIR}/files/"*.yaml 2>/dev/null) ]]; then
build_modules_yaml "$@"
else
build_modules_legacy "$@"
fi
}
#.............................................................................
# main
parse_args "$@"
if [[ -z "${opt_system}" ]]; then
opt_system=$(std::get_os_release)
fi
opt_system="${opt_system:-$(std::get_os_release)}"
pbuild.jobs "${opt_jobs}"
pbuild.force_rebuild "${opt_force_rebuild}"
@@ -416,20 +674,17 @@ pbuild.update_modulefiles "${opt_update_modulefiles}"
pbuild.system "${opt_system}"
pbuild.verbose "${opt_verbose}"
if [[ "${opt_bootstrap}" == 'yes' ]]; then
test -d "${BUILDBLOCK_DIR}/../../config" && PATH+=":$_"
else
# Please note: if we trap DEBUG a statement like
# test -d ... && PATH+=$_
# does not work (at least on macOS with bash 4 and 5)
if [[ -d "${PMODULES_ROOT}/${PMODULES_CONFIG_DIR}" ]]; then
PATH+=":${PMODULES_ROOT}/${PMODULES_CONFIG_DIR}"
fi
fi
source "${opt_build_config}" || \
std::die 3 "Oops: Cannot source configuration file -- '$_'"
#
# read configuration for modbuild
#
pm::read_config
: ${PMODULES_DISTFILESDIR:=${PMODULES_ROOT}/var/distfiles}
# :FIXME: should dist files go to
# ${pm_root}/var/distfiles
# or
# ${overlay}/var/distfiles
# ?
: ${PMODULES_DISTFILESDIR:="${PMODULES_HOME%%/Tools*}/var/distfiles"}
: ${PMODULES_TMPDIR:=/var/tmp/${USER}}
declare -r BUILD_SCRIPT
@@ -439,17 +694,6 @@ declare -r BUILDBLOCK_DIR
IFS=/ read -r -a fname <<< "${BUILD_SCRIPT:1}"
module_name=${fname[${#fname[@]}-2]}
#
# are we bootstrapping? If yes, go for it...
#
if [[ "${opt_bootstrap}" == 'yes' ]]; then
pbuild.bootstrap "${module_name}" "${versions[0]}" 'stable'
exit $?
fi
#
# else
#
for version in "${versions[@]}"; do
build_modules "${module_name}" "${version}" "${opt_with_modules[@]}"
done
+1006 -526
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+8 -4
View File
@@ -10,9 +10,8 @@ Pmodules are a hierarchical module environment based on Environment Modules.
"
#
# It might be that '${PMODULES_ROOT}/Tools/Pmodules/VERSION' is in PATH.
# Why? With older version the PATH might have been set without loading
# a module.
# Older versions add '$PREFIX/bin' to PATH without loading Pmodules as
# a module. Remove it ...
#
if { [module-info mode load] } {
set PATH ":$::env(PATH):"
@@ -21,4 +20,9 @@ if { [module-info mode load] } {
remove-path PATH $str
}
}
}
remove-path C_INCLUDE_PATH "$PREFIX/include"
remove-path CPLUS_INCLUDE_PATH "$PREFIX/include"
set shell [module-info shell]
puts "source \"$PREFIX/init/$shell\""
}
+6 -4
View File
@@ -1,13 +1,15 @@
#!/bin/bash
#
# The following settings are system defaults. They can be (re-)defined
# in a system wide profile or in a user's profile.
# set default version
: ${PMODULES_VERSION:=@PMODULES_VERSION@}
export PMODULES_VERSION
declare -x PMODULES_ROOT=$(cd $(dirname "${BASH_SOURCE}")/.. && pwd)
declare -x PMODULES_HOME="${PMODULES_ROOT}/Tools/Pmodules/${PMODULES_VERSION}"
##### no changes below this line ######
declare __pm_root__=$(cd $(dirname "${BASH_SOURCE}")/.. && pwd)
declare -x PMODULES_HOME="${__pm_root__}/Tools/Pmodules/${PMODULES_VERSION}"
unset __pm_root__
test -r "${PMODULES_HOME}/init/bash" && source "$_"
+1 -21
View File
@@ -9,28 +9,8 @@ if ( ${PMODULES_VERSION} == "" ) then
setenv PMODULES_VERSION "@PMODULES_VERSION@"
endif
#############################################################################
# N O C H A N G E S B E L O W T H I S L I N E ! #
#############################################################################
setenv PMODULES_HOME "@PMODULES_ROOT@/Tools/Pmodules/${PMODULES_VERSION}"
setenv PMODULES_MODULEFILES_DIR 'modulefiles'
setenv ReleaseStages ':unstable:stable:deprecated:'
setenv PMODULES_ROOT "@PMODULES_ROOT@"
setenv PMODULES_CONFIG_DIR 'config'
setenv PMODULES_HOME "${PMODULES_ROOT}/Tools/Pmodules/${PMODULES_VERSION}"
setenv PMODULES_DIR "${PMODULES_HOME}"
############################################################################
# some sanity checks
#
# fixme: add some checks
############################################################################
# inititialize Pmodules for csh
#
set _init_csh="${PMODULES_HOME}/init/csh"
if ( ! -r "${_init_csh}" ) then
echo "Oops: cannot initialize Modules!"
+10 -40
View File
@@ -1,50 +1,20 @@
#!/bin/bash
#
# Notes:
# - PMODULES_ROOT is derived from the location of this file.
# - Some for PMODULES_CONFIG_DIR.
# - The Pmodules software must be installed in
# ${PMODULES_ROOT}/Tools/Pmodules/${PMODULES_VERSION}
#!/bin/zsh
#
declare -xa DefaultGroups
declare -xa DefaultReleaseStages
declare -x PMODULES_VERSION
: ${DefaultGroups:=(Tools Programming)}
: ${DefaultReleaseStages:=(stable)}
# set default version
: ${PMODULES_VERSION:=@PMODULES_VERSION@}
#############################################################################
# N O C H A N G E S B E L O W T H I S L I N E ! #
#############################################################################
##### no changes below this line ######
export PMODULES_VERSION
declare -x PMODULES_MODULEFILES_DIR='modulefiles'
declare -x ReleaseStages=':unstable:stable:deprecated:'
declare __this_file__=${(%):-%N}
declare -x PMODULES_HOME="${__this_file____:h:h}/Tools/Pmodules/${PMODULES_VERSION}"
unset __this_file__
declare -x PMODULES_ROOT=$(cd $(dirname "${(%):-%N}")/.. && pwd)
declare -x PMODULES_CONFIG_DIR=$(basename $(cd $(dirname "${(%):-%N}") && pwd))
declare -x PMODULES_HOME="${PMODULES_ROOT}/Tools/Pmodules/${PMODULES_VERSION}"
declare -x PMODULES_DIR="${PMODULES_HOME}"
############################################################################
# some sanity checks
#
if [[ ! -d ${PMODULES_ROOT} ]]; then
echo "Oops: ${PMODULES_ROOT}: Set as Pmodules root, but this is not a directory." 1>&2
return 1
fi
if [[ ! -d ${PMODULES_HOME} ]]; then
echo "Oops: ${PMODULES_HOME}: Set as Pmodules home, but this is not a directory." 1>&2
return 1
fi
############################################################################
# inititialize Pmodules for zsh
#
test -r "${PMODULES_HOME}/init/zsh" && source "$_"
if (( $? != 0 )); then
echo "Oops: cannot initialize Pmodules environment!"
if [[ $? != 0 ]]; then
echo "Oops: cannot initialize Pmodules!"
return 1
fi
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
+30 -96
View File
@@ -1,105 +1,19 @@
#!/bin/zsh
#############################################################################
# implement module comand as function
############################################################################
# some sanity checks
#
module() {
local -r modulecmd="${PMODULES_HOME}/bin/modulecmd"
local -a args=()
local -a modulecmd_opts=()
local -a subcmd_opts=()
while (( $# > 0 )); do
case $1 in
-* )
modulecmd_opts+=( $1 )
shift
;;
* )
break
;;
esac
done
local -r subcmd="$1"
shift
while (( $# > 0 )); do
case $1 in
-* )
subcmd_opts+=( $1 )
;;
[/~a-zA-Z]* )
args+=( $1 )
;;
esac
shift
done
[[ -z ${subcmd} ]] && args+=( 'help' )
[[ ${#args[@]} == 0 ]] && args+=( '--' )
# Loop over all modules to load.
# Note: We have to eval here, otherwise we cannot do something like
# $ module load gcc/5.2.0 openmpi/1.8.8 hdf5/1.8.15
local m
for module in ${args[@]}; do
eval $( "${modulecmd}" zsh "${modulecmd_opts[@]}" "${subcmd}" \
"${subcmd_opts[@]}" "${module}" )
done
}
export -f module
if [[ ! -d ${PMODULES_HOME} ]]; then
echo "Oops: ${PMODULES_HOME}: Set as Pmodules home, but this is not a directory." 1>&2
return 1
fi
#############################################################################
# helper functions
# initialize zsh completion
#
save_env() {
local s=''
while (( $# > 0 )); do
s+="$( typeset -p $1 );"
shift
done
echo export PMODULES_ENV=$( "${PMODULES_HOME}/sbin/base64" --wrap=0 <<< "$s" )
}
#############################################################################
# setup environment
#
declare -x LOADEDMODULES=''
declare -x _LMFILES_=''
# build initial MODULEPATH
declare -x MODULEPATH=''
typeset -T MODULEPATH modulepath
for group in ${DefaultGroups[@]}; do
dir="${PMODULES_ROOT}/${group}/${PMODULES_MODULEFILES_DIR}"
modulepath=( "${dir}" ${(m)modulepath:#${dir}} )
done
# build initial list of used releases
declare -x UsedReleases=''
typeset -T UsedReleases usedreleases
for r in ${DefaultReleaseStages[@]}; do
usedreleases=( "${r}" ${(m)usedreleases:#${r}} )
done
eval $(save_env UsedReleases DefaultReleaseStages DefaultGroups ReleaseStages)
unset UsedReleases
unset DefaultReleaseStages
unset DefaultGroups
unset ReleaseStages
# initialize MANPATH with output of `man --path` if not set
[[ -z "${MANPATH}" ]] && manpath=$( man --path )
# add man pages of Pmodules and remove all other Pmodules man pages
manpath=( "${PMODULES_HOME}/share/man" ${(m)manpath:#${dirs_to_remove}} )
# initialize completion
test -r "${PMODULES_HOME}/init/zsh_completion" && source "$_"
# cleanup
unset group dir r dirs_to_remove
#if [[ -r "${PMODULES_HOME}/init/bash_completion" ]]; then
# source "${PMODULES_HOME}/init/bash_completion"
#fi
#############################################################################
# legacy...
@@ -107,6 +21,26 @@ unset group dir r dirs_to_remove
unset MODULE_VERSION
unset MODULE_VERSION_STACK
unset MODULESHOME
unset PMODULES_ENV
declare -x PMODULES_DIR="${PMODULES_HOME}"
#############################################################################
# implement module comand as shell function
#
module() {
eval $("${PMODULES_HOME}/bin/modulecmd" bash "$@")
}
export -f module
modbuild(){
(
eval $("${PMODULES_HOME}/bin/modulecmd" bash load System:bash)
"${PMODULES_HOME}/bin/modbuild" "$@"
)
}
export -f modbuild
# Local Variables:
# mode: sh
-25
View File
@@ -1,25 +0,0 @@
#!/usr/bin/env modbuild
pbuild::set_download_url "ftp://ftp.tcl.tk/pub/tcl/tcl8_6/tcl$V-src.tar.gz"
pbuild::install_docfiles 'license.terms' 'README.md'
pbuild::configure() {
case ${OS} in
Linux )
srcdir="${SRC_DIR}/unix"
;;
Darwin )
srcdir="${SRC_DIR}/macosx"
;;
esac
"${srcdir}"/configure \
--prefix="${PREFIX}" \
--bindir="${PREFIX}/${UTILBIN_DIR}" \
--enable-shared=no \
|| exit 1
}
pbuild::post_install() {
{ cd "${PREFIX}/${UTILBIN_DIR}" && rm -f tclsh && ln -fs tclsh${V%.*} tclsh; };
}
-5
View File
@@ -1,5 +0,0 @@
#!/usr/bin/env modbuild
pbuild::set_download_url "https://ftp.gnu.org/gnu/$P/$P-$V.tar.gz"
pbuild::add_configure_args "--bindir=${PREFIX}/${UTILBIN_DIR}"
-4
View File
@@ -1,4 +0,0 @@
#!/usr/bin/env modbuild
pbuild::set_download_url "http://ftp.gnu.org/gnu/coreutils/$P-$V.tar.xz"
pbuild::add_configure_args "--bindir=${PREFIX}/${UTILBIN_DIR}"
-10
View File
@@ -1,10 +0,0 @@
#!/usr/bin/env pbuild
pmodules.configure() {
"${MODULE_SRCDIR}"/configure \
--prefix="${PREFIX}" \
|| exit 1
}
pmodules.add_to_group 'Tools'
pmodules.make_all
-4
View File
@@ -1,4 +0,0 @@
#!/usr/bin/env modbuild
pbuild::set_download_url "https://ftp.gnu.org/pub/gnu/$P/$P-$V.tar.xz"
pbuild::add_configure_args "--bindir=${PREFIX}/${UTILBIN_DIR}"
-12
View File
@@ -1,12 +0,0 @@
diff -u getopt-1.1.6.orig//Makefile getopt-1.1.6/Makefile
--- getopt-1.1.6.orig//Makefile 2014-11-24 04:33:39.000000000 -0800
+++ getopt-1.1.6/Makefile 2016-10-27 08:27:52.000000000 -0700
@@ -61,7 +61,7 @@
-$(RM) $(objects) $(binaries)
getopt: $(objects)
- $(CC) $(LDFLAGS) -o $@ $(objects)
+ $(CC) -o $@ $(objects) $(LDFLAGS)
install: getopt install_po
$(INSTALL) -m 755 -d $(DESTDIR)$(bindir) $(DESTDIR)$(man1dir)
-54
View File
@@ -1,54 +0,0 @@
#!/usr/bin/env modbuild
set -x
pbuild::set_download_url "http://frodo.looijaard.name/system/files/software/getopt/getopt-1.1.6.tar.gz"
pbuild::compile_in_sourcetree
pbuild::pre_prep_Linux() {
pbuild::add_patch "Makefile.patch"
}
pbuild::configure() {
:
}
pbuild::compile() {
declare -x C_INCLUDE_PATH="${PREFIX}/include"
declare -x LIBRARY_PATH="${PREFIX}/lib"
case ${OS} in
Linux )
declare -x C_INCLUDE_PATH="${PREFIX}/include"
declare -x LIBRARY_PATH="${PREFIX}/lib"
declare -x LDFLAGS="-lintl -L${PREFIX}/lib"
;;
Darwin )
if [[ ! -d '/opt/local/bin' ]] || [[ ! -x '/opt/local/bin/msgfmt' ]]; then
std::die 1 "gettext port from Macports is required to build 'getopt'!"
fi
PATH+=':/opt/local/bin'
declare -x C_INCLUDE_PATH='/opt/local/include'
declare -x LDFLAGS="/opt/local/lib/libintl.a /opt/local/lib/libiconv.a -framework CoreFoundation"
;;
esac
make -e all || exit 1
declare -x DESTDIR="${PREFIX}"
declare -x prefix='' || exit 1
PATH="${PREFIX}/${UTILBIN_DIR}:${PATH}"
make -e install
mv "${PREFIX}/bin/getopt" "${PREFIX}/${UTILBIN_DIR}"
}
pbuild::install() {
:
}
pbuild::cleanup_build() {
:
}
# Local Variables:
# mode: sh
# sh-basic-offset: 8
# tab-width: 8
# End:
-19
View File
@@ -1,19 +0,0 @@
#!/usr/bin/env modbuild
pbuild::set_download_url "https://ftp.gnu.org/pub/gnu/$P/$P-$V.tar.gz"
pbuild::add_configure_args "--prefix=${PREFIX}"
pbuild::add_configure_args "--bindir=${PREFIX}/${UTILBIN_DIR}"
pbuild::add_configure_args "--disable-java"
pbuild::add_configure_args "--disable-threads"
pbuild::add_configure_args "--disable-shared"
pbuild::add_configure_args "--enable-relocatable"
pbuild::add_configure_args "--disable-openmp"
pbuild::add_configure_args "--disable-acl"
pbuild::add_configure_args "--disable-curses"
pbuild::add_configure_args "--with-included-gettext"
pbuild::add_configure_args "--without-libiconv-prefix"
pbuild::add_configure_args "--without-libintl-prefix"
pbuild::add_configure_args "--with-included-libxml"
pbuild::add_configure_args "--with-pic=yes"
-56
View File
@@ -1,56 +0,0 @@
#!/usr/bin/env modbuild
# use system gcc to compile
declare -rx CC=gcc
pbuild::set_download_url "https://amas.web.psi.ch/Downloads/$P/$P-$V.tar.bz2"
TCL_DIR="${PMODULES_ROOT}/Tools/Pmodules/${PMODULES_VERSION}"
PATH="${TCL_DIR}/bin:${PATH}"
pbuild::configure() {
case ${OS} in
Linux )
declare -x LIBS="-lz -lpthread"
;;
Darwin )
declare -x LIBS="-lz -framework CoreFoundation"
;;
esac
CPPFLAGS="-DUSE_INTERP_ERRORLINE" "${SRC_DIR}"/configure \
--prefix="${PREFIX}" \
--exec-prefix="${PREFIX}" \
--with-module-path="${PMODULES_ROOT}/Tools/${PMODULES_MODULEFILES_DIR}" \
--with-tcl="${TCL_DIR}/lib" \
--without-x \
--disable-versioning \
|| exit 1
}
pbuild::post_install() {
rm -v "${PREFIX}/Modules/bin/add.modules"
rm -v "${PREFIX}/Modules/bin/mkroot"
rm -rfv "${PREFIX}/Modules/modulefiles"
mv -v "${PREFIX}/Modules/share/man/man1/module.1 ${PREFIX}/share/man/man1"
mv -v "${PREFIX}/Modules/share/man/man4/modulefile.4 ${PREFIX}/share/man/man4"
rmdir "${PREFIX}/Modules/bin"
rmdir "${PREFIX}/Modules/share/man/man1"
rmdir "${PREFIX}/Modules/share/man/man4"
rmdir "${PREFIX}/Modules/share/man"
rmdir "${PREFIX}/Modules/share"
rmdir "${PREFIX}/Modules"
rm -f "${PREIX}/init/{ksh,perl.pm,python.py,ruby.rb,cmake,.modulespath}"
cp -v "${BUILD_DIR}/modulecmd" "${PREFIX}/libexec/modulecmd.bin" || exit 1
}
# fake module command
module() {
:
}
# Local Variables:
# mode: sh
# sh-basic-offset: 8
# tab-width: 8
# End:
-5
View File
@@ -1,5 +0,0 @@
#!/usr/bin/env modbuild
pbuild::set_download_url "https://core.tcl-lang.org/tcllib/uv/$P-$V.tar.xz"
+308 -331
View File
@@ -1,48 +1,84 @@
#!/usr/bin/env bash
#
if [ "${BASH_VERSINFO[0]}" -lt 4 ]; then
echo "BASH version >= 4 is required and must be available in PATH!" 1>&2
declare -rx VERSION='1.1.10'
if (( "${BASH_VERSINFO[0]}" < 5 )); then
echo "BASH version 5.0 or newer is required and must be available in PATH!" 1>&2
exit 1
fi
declare -r OS=$(uname -s)
set -o nounset
set -o pipefail
shopt -s nullglob
declare BOOTSTRAP_DIR=$(dirname "$0")
source "${BOOTSTRAP_DIR}/Pmodules/libstd.bash" || { echo "Oops!" 1>&2; exit 42; }
declare -r BOOTSTRAP_DIR=$(std::get_abspath "${BOOTSTRAP_DIR}")
declare -r BOOTSTRAP_DIR="$(cd "$(dirname "$0")" && pwd -P)"
declare -r SRC_DIR="${BOOTSTRAP_DIR}/Pmodules"
# these variables need to be defined but must be initialized with an empty string
declare -x PMODULES_HOME=''
declare -x PMODULES_DISTFILESDIR=''
declare -x PMODULES_TMPDIR=''
source "${SRC_DIR}/libstd.bash" || { echo "Oops!" 1>&2; exit 42; }
source "${SRC_DIR}/libpbuild.bash" || { echo "Oops!" 1>&2; exit 42; }
declare -r PMOD_DIR="Tools/Pmodules/${VERSION}"
# config directory and file relative to install root
declare -rx CONFIG_DIR='config'
declare -rx CONFIG_FILE='modbuild.conf'
# defaults
declare -rx DEFAULT_PMODULES_ROOT='/opt/psi'
declare -rx DEFAULT_DISTFILES_DIR="var/distfiles"
declare -rx DEFAULT_VERSIONS_CONFIG="${CONFIG_DIR}/versions.conf"
declare -rx DEFAULT_TMPDIR='var/tmp/${USER}'
declare -rx CONFIG_FILE="${CONFIG_DIR}/Pmodules.yaml"
# directory where the required tools will be installed (like bash, tclsh, etc)
declare -rx UTILBIN_DIR='libexec'
# defaults
declare -rx DEFAULT_INSTALL_ROOT='/opt/psi'
declare -rx DEFAULT_DISTFILES_DIR='var/distfiles'
declare -rx DEFAULT_TMP_DIR='var/tmp/${USER}'
#-----------------------------------------------------------------------------
#
get_version() {
local -r name="$1"
echo $(awk "/^$1[[:blank:]]/ {print \$2}" "${DEFAULT_VERSIONS_CONFIG}")
read_config_file() {
local fname="$1"
if [[ ! -r "${fname}" ]]; then
std::die 1 "Configuration file '${fname}' does not exist or is not readable!"
fi
eval $(std::parse_yaml "${fname}" '') || \
std::die 1 "Cannot read configuration file '${fname}'"
declare -xg INSTALL_ROOT="${Overlays_base_install_root}"
if [[ -z "${INSTALL_ROOT}" ]]; then
std::die 1 "Error in configuration file '${fname}': install root not defined!"
fi
declare -xg PREFIX="${INSTALL_ROOT}/${PMOD_DIR}"
declare -xg DOWNLOADS_DIR="${DistfilesDir:-${INSTALL_ROOT}/${DEFAULT_DISTFILES_DIR}}"
declare -xg TMP_DIR="${TmpDir:-${INSTALL_ROOT}/${DEFAULT_TMP_DIR}}"
}
declare -rx PMODULES_VERSION=$(get_version 'Pmodules')
#-----------------------------------------------------------------------------
# The next functions are used in the sub-commands, if an illegal option
# or argument has been passed.
#
illegal_option(){
local subcmd="$1"
local opt="$2"
std::die 1 \
"%s: %s -- %s" \
"$(basename $0) ${subcmd}" \
"Illegal option" \
"${opt}"
}
illegal_arg(){
local subcmd="$1"
local arg="$2"
std::die 1 \
"%s: %s -- %s" \
"$(basename $0) ${subcmd}" \
"Illegal argument" \
"${arg}"
}
#-----------------------------------------------------------------------------
#
usage() {
local prog=$(basename $0)
# help for sub-command 'help' (usage)
build::help_help(){
local prog="$(basename "$0")"
echo "
Usage: ${prog} help|configure|compile|install
@@ -65,59 +101,36 @@ to get help for a specific sub-command.
}
#-----------------------------------------------------------------------------
# sub-command 'help'
#
pmodules::help() {
if (( $# > 1 )); then
usage
# print help for sub-commands
#
build::help() {
if (( $# == 0 )); then
build::help_help
else
case $1 in
configure|compile|install )
build::help_$1
;;
help )
build::help_help
;;
-* )
illegal_option 'help' "$1"
;;
* )
std::error "No such command -- $1"
build::help_help
;;
esac
fi
case $1 in
configure|compile|install )
pmodules::help_$1
;;
* )
echo -en "$1 - invalid sub-command!\n" 1>&2
usage
;;
esac
}
#-----------------------------------------------------------------------------
# help for sub-command 'configure'
#
read_config_file() {
local var="$1"
local fname="${!var}"
if [[ ! -r "${fname}" ]]; then
std::die 1 "Configuration file '${fname}' does not exist or is not readable!"
fi
source "${fname}" || std::die 1 "Cannot read configuration file '${fname}'"
if [[ -z "${PMODULES_ROOT}" ]]; then
std::die 1 "Error in configuration file '${fname}': PMODULE_ROOT not defined!"
fi
if [[ -z "${PMODULES_VERSION}" ]]; then
std::die 1 "Error in configuration file '${fname}': PMODULE_VERSION not defined!"
fi
if [[ -z "${PMODULES_DISTFILESDIR}" ]]; then
std::die 1 "Error in configuration file '${fname}': PMODULES_DISTFILESDIR not defined!"
fi
if [[ -z "${PMODULES_TMPDIR}" ]]; then
std::die 1 "Error in configuration file '${fname}': PMODULE_TMPDIR not defined!"
fi
# we have to make these variables global, otherwise they are not set
# in the calling function
declare -g PMODULES_ROOT="${PMODULES_ROOT}"
declare -g PMODULES_HOME="${PMODULES_HOME}"
declare -g PMODULES_DISTFILESDIR="${PMODULES_DISTFILESDIR}"
declare -g PMODULES_TMPDIR="${PMODULES_TMPDIR}"
std::upvar "${var}" "${fname}"
}
#-----------------------------------------------------------------------------
#
pmodules::help_configure() {
build::help_configure() {
echo "
Usage: $(basename $0) configure [OPTION...]
@@ -125,21 +138,24 @@ Configure and setup a new Pmodules environment. You need permissions
to write to the installation root.
Options:
--prefix=DIR
--install_root=DIR
Root of the Pmodules environment installation. Everything will be
installed in a directory hierarchy with 'DIR' as prefix.
The default is '${DEFAULT_PMODULES_ROOT}'.
The default is '${DEFAULT_INSTALL_ROOT}'.
--distfilesdir=DIR
Directory where downloaded files are stored.
The default is '${DEFAULT_PMODULES_DISTFILESDIR}' in the
The default is '${DEFAULT_INSTALL_ROOT}/${DEFAULT_DISTFILES_DIR}' in the
Pmodules root directory.
--tmpdir=DIR
Directory for temporary files.
The default is '${DEFAULT_PMODULES_TMPDIR}'
The default is '${DEFAULT_INSTALL_ROOT}/${DEFAULT_TMP_DIR}'
--help
--force|-f
Override existing configuration.
--help|-h|-?
Print this help text.
" 1>&2
@@ -147,102 +163,123 @@ Options:
}
#-----------------------------------------------------------------------------
# sub-command 'configure'
#
pmodules::configure() {
local prefix="${PMODULES_ROOT:-${DEFAULT_PMODULES_ROOT}}"
local distfilesdir=''
local tmpdir=''
local config_file=''
# Create basic directory hierachy and the configuration file in the given
# installation root directory
#
build::configure() {
local opt_force='no'
while (( $# > 0 )); do
case "$1" in
--prefix )
prefix="$2"
shift 1
;;
--prefix=* )
prefix="${1#*=}"
--install_root | --install_root=* )
if [[ $1 == *=* ]]; then
INSTALL_ROOT="${1#*=}"
else
INSTALL_ROOT="$2"
shift 1
fi
;;
--distfilesdir )
distfilesdir="$2"
shift 1
;;
--distfilesdir=* )
distfilesdir="${1#*=}"
if [[ $1 == *=* ]]; then
DOWNLOADS_DIR="${1#*=}"
else
DOWNLOADS_DIR="$2"
shift 1
fi
;;
-f | --force )
opt_force='yes'
;;
--tmpdir )
tmpdir="$2"
shift 1
;;
--tmpdir=* )
tmpdir="${1#*=}"
--tmpdir | --tmpdir=* )
if [[ $1 == *=* ]]; then
TMP_DIR="${1#*=}"
else
TMP_DIR="$2"
shift 1
fi
;;
--help | -h | -\? )
pmodules::help_configure
build::help_configure
;;
-* )
std::die 1 "$1: illegal option"
illegal_option 'configure' "$1"
;;
* )
std::die 1 "$1: illegal argument to sub-command 'configure'."
illegal_arg 'configure' "$1"
;;
esac
shift 1
done
if [[ ! -d ${prefix} ]]; then
echo "The root directory '${prefix}' does not exist, trying to create it..."
if ! mkdir -p "${prefix}"; then
std::die 1 "Creating the root directory failed!\nAborting..."
: ${INSTALL_ROOT:=${DEFAULT_INSTALL_ROOT}}
: ${DOWNLOADS_DIR:=${INSTALL_ROOT}/${DEFAULT_DISTFILES_DIR}}
: ${TMP_DIR:=${INSTALL_ROOT}/${DEFAULT_TMP_DIR}}
PREFIX="${INSTALL_ROOT}/${PMOD_DIR}"
#---
# check/create the install root
if [[ ! -d ${INSTALL_ROOT} ]]; then
std::info "%s\n%s" \
"The root directory '${INSTALL_ROOT}' does not exist!" \
"Trying to create it..."
if ! mkdir -p "${INSTALL_ROOT}"; then
std::die 1 "%s\n%s" \
"Creating the root directory failed!" \
"Aborting..."
fi
fi
if [[ ! -w ${prefix} ]]; then
std::die 1 "The root directory '${prefix}' is not writable!\nAborting..."
if [[ ! -w ${INSTALL_ROOT} ]]; then
std::die 1 "%s\n%s" \
"The root directory '${INSTALL_ROOT}' is not writable!" \
"Aborting..."
fi
mkdir -p "${prefix}/${CONFIG_DIR}" || \
#---
# check/create YAML config file in install root
mkdir -p "${INSTALL_ROOT}/${CONFIG_DIR}" || \
std::die 1 "Aborting..."
local config_file="${prefix}/${CONFIG_DIR}/${CONFIG_FILE}"
if [[ "${opt_force}" == 'yes' ]]; then
rm -f "${config_file}"
local config_file="${INSTALL_ROOT}/${CONFIG_FILE}"
if [[ "${opt_force}" != 'yes' ]] && [[ -e "${config_file}" ]]; then
std::die 1 "%s\n%s" \
"The Pmodules environment in '${INSTALL_ROOT}' has already been configured!" \
"Use the option --force to override. Aborting..."
fi
if [[ -e "${config_file}" ]]; then
std::die 1 "The Pmodules environment has already been configured!
Use the option --force to override.\nAborting..."
fi
[[ -z "${distfilesdir}" ]] && distfilesdir="${prefix}/${DEFAULT_DISTFILES_DIR}"
[[ -z "${tmpdir}" ]] && tmpdir="${prefix}/${DEFAULT_TMPDIR}"
sed_cmd="s:@PMODULES_ROOT@:${prefix}:g;"
sed_cmd+="s:@PMODULES_DISTFILESDIR@:${distfilesdir}:g;"
sed_cmd+="s:@PMODULES_TMPDIR@:${tmpdir}:g;"
sed_cmd+="s:@PMODULES_VERSION@:${PMODULES_VERSION}:g"
sed_cmd="s:@INSTALL_ROOT@:${INSTALL_ROOT}:g;"
sed_cmd+="s:@PMODULES_DISTFILESDIR@:${DOWNLOADS_DIR}:g;"
sed_cmd+="s:@PMODULES_TMPDIR@:${TMP_DIR}:g;"
sed_cmd+="s:@PMODULES_VERSION@:${VERSION}:g"
sed "${sed_cmd}" "${BOOTSTRAP_DIR}/${CONFIG_DIR}/${CONFIG_FILE}.in" \
sed "${sed_cmd}" "${BOOTSTRAP_DIR}/${CONFIG_FILE}.in" \
> "${config_file}" || \
std::die 1 "Cannot create configuration file in Pmodules root\nAborting..."
read_config_file 'config_file'
install -d -m 0755 "${PMODULES_HOME}/bin"
install -d -m 0755 "${PMODULES_HOME}/init"
install -d -m 0755 "${PMODULES_HOME}/lib"
install -d -m 0755 "${PMODULES_HOME}/libexec"
#---
# create basic directories
install -d -m 0755 \
"${INSTALL_ROOT}/Tools/modulefiles/Pmodules" \
"${INSTALL_ROOT}/Libraries/modulefiles" \
"${INSTALL_ROOT}/Programming/modulefiles" \
"${DOWNLOADS_DIR}" || \
std::die 1 "%s" \
"Creating basic directories failed\n" \
"Aborting..."
#---
echo "Configuration:"
echo " root of Pmodules environment: ${prefix}"
echo " Pmodule prefix: ${PMODULES_HOME}"
echo " tmp directory: ${tmpdir}"
echo " store for downloaded files: ${distfilesdir}"
echo " root of Pmodules environment: ${INSTALL_ROOT}"
echo " Pmodule prefix: ${PREFIX}"
echo " tmp directory: ${TMP_DIR}"
echo " store for downloaded files: ${DOWNLOADS_DIR}"
echo "Done..."
}
#-----------------------------------------------------------------------------
# help for sub-command 'compile'
#
pmodules::help_compile() {
build::help_compile() {
echo "
Usage: $(basename $0) compile [OPTION...]
@@ -250,22 +287,14 @@ Compile and install the required tools for a new Pmodules environment.
ou need the permissions to write to the installation root.
Options:
--prefix=DIR
Root of the Pmodules environment installation. The root of the
installation must be either specified via this option or the
environment variable PMODULES_ROOT. If this option is used and
the PMODULES_ROOT is set, the directory specified with this
option will be used.
--install_root=DIR
Root of the Pmodules environment installation. Everything will be
installed in a directory hierarchy with 'DIR' as prefix.
The default is '${DEFAULT_INSTALL_ROOT}'.
--debug
Enable verbose/debug output.
--disable-cleanup )
--disable-cleanup
Do not cleanup the tmp directory after compilation and installation.
--force | -f
Force compilation.
--help
Print this help text.
@@ -273,266 +302,208 @@ Options:
std::die 1 ""
}
pmodules::compile() {
build () {
local -r name="$1"
local -r version=$(get_version "${name}")
shift
"${BOOTSTRAP_DIR}/Pmodules/modbuild.in" \
"--config=${config_file}" \
"--enable-cleanup" \
"--force-rebuild" \
"--debug" \
"--verbose" \
"${BOOTSTRAP_DIR}/Tools/${name}/build" \
"${build_opts[@]}" "$@" "${version}" || \
std::die 3 "Compiling '${name}' failed!"
}
local prefix="${PMODULES_ROOT:-${DEFAULT_PMODULES_ROOT}}"
#-----------------------------------------------------------------------------
# sub-command 'compile'
#
# compile all required tools like bash, tclsh etc for a Pmodules module.
# The version is defined at the beginning of this file.
#
build::compile() {
local opt_force='no'
local config_file=''
local opt_cleanup='yes'
while (( $# > 0 )); do
case $1 in
--prefix )
prefix="$2"
shift 1
;;
--prefix=* )
prefix="${1#*=}"
--install_root | --install_root=* )
if [[ $1 == *=* ]]; then
INSTALL_ROOT="${1#*=}"
else
INSTALL_ROOT="$2"
shift 1
fi
;;
--disable-cleanup )
build_opts+=( "$1" )
;;
--debug )
build_opts+=( "$1" )
;;
-f | --force )
opt_force='yes'
opt_cleanup='no'
;;
--help | -h | -\? )
pmodules::help_compile
build::help_compile
;;
-* )
std::die 1 "$1: illegal option"
illegal_option 'compile' "$1"
;;
* )
std::die 1 "$1: illegal argument for sub-command 'compile'."
illegal_arg 'compile' "$1"
;;
esac
shift 1
done
: ${INSTALL_ROOT:=${DEFAULT_INSTALL_ROOT}}
PREFIX="${INSTALL_ROOT}/${PMOD_DIR}"
local config_file="${prefix}/${CONFIG_DIR}/${CONFIG_FILE}"
read_config_file config_file
install -d -m 0755 "${PMODULES_HOME}/bin"
install -d -m 0755 "${PMODULES_HOME}/init"
install -d -m 0755 "${PMODULES_HOME}/lib"
install -d -m 0755 "${PMODULES_HOME}/libexec"
read_config_file "${INSTALL_ROOT}/${CONFIG_FILE}"
echo "Configuration:"
echo " root of Pmodules environment: ${prefix}"
echo " Pmodule prefix: ${PMODULES_HOME}"
echo " root of Pmodules environment: ${INSTALL_ROOT}"
echo " Pmodule prefix: ${PREFIX}"
#if [[ ! -f "${PMODULES_HOME}/${UTILBIN_DIR}/base64" ]] || [[ ${opt_force} == 'yes' ]]; then
# build coreutils
#fi
install -m 0755 -d "${PREFIX}"/{bin,init,lib,libexec} \
if [[ "${OS}" == 'Darwin' ]]; then
if [[ ! -f "${PMODULES_HOME}/${UTILBIN_DIR}/getopt" ]] || [[ ${opt_force} == 'yes' ]]; then
build getopt
fi
if [[ ! -f "${PMODULES_HOME}/${UTILBIN_DIR}/find" ]] || [[ ${opt_force} == 'yes' ]]; then
build findutils
fi
for recipe in recipes/[0-9]*; do
"./${recipe}" "${PREFIX}"
done
if [[ "${opt_cleanup}" == 'yes' ]]; then
rm -rf "${TMP_DIR}/*"
rm -f "${PREFIX}/lib/libtcl*.a"
rm -rf "${PREFIX}/include"
fi
if [[ ! -f "${PMODULES_HOME}/${UTILBIN_DIR}/bash" ]] || [[ ${opt_force} == 'yes' ]]; then
build bash
fi
if [[ ! -e "${PMODULES_HOME}/${UTILBIN_DIR}/tclsh" ]] || [[ ${opt_force} == 'yes' ]]; then
build Tcl
fi
if [[ ! -e "${PMODULES_HOME}/lib/tcllib1.20" ]] || [[ ${opt_force} == 'yes' ]]; then
build tcllib
fi
if [[ ! -e "${PMODULES_HOME}/libexec/modulecmd.bin" ]] || [[ ${opt_force} == 'yes' ]]; then
build modules
fi
rm -rf "${PMODULES_HOME}/include"
rm -rf "${PMODULES_HOME}/lib/"*.a
rm -rf "${PMODULES_HOME}/lib/"*.la
rm -rf "${PMODULES_HOME}/lib/bash"
rm -rf "${PMODULES_HOME}/lib/pkginfo"
rm -rf "${PMODULES_HOME}/man"
rm -rf "${PMODULES_HOME}/share"
echo "Done..."
}
#-----------------------------------------------------------------------------
# help for sub-command 'install'
#
pmodules::help_install() {
build::help_install() {
echo "
Usage: $(basename $0) install [OPTION...]
Install a new Pmodules version.
Options:
--prefix=DIR
Root of the Pmodules environment installation. The root of the
installation must be either specified via this option or the
environment variable PMODULES_ROOT. If this option is used and
the PMODULES_ROOT is set, the directory specified with this
option will be used.
--install_root=DIR
Root of the Pmodules environment installation. Everything will be
installed in a directory hierarchy with 'DIR' as prefix.
The default is '${DEFAULT_INSTALL_ROOT}'.
--debug
Enable verbose/debug output.
--disable-cleanup )
Do not cleanup the tmp directory after compilation and installation.
--force | -f
Force compilation.
--help
--help|-h|-?
Print this help text.
" 1>&2
std::die 1 ""
}
pmodules::install() {
local prefix="${PMODULES_ROOT:-${DEFAULT_PMODULES_ROOT}}"
local config_file=''
local opt_force='no'
#-----------------------------------------------------------------------------
# sub-command 'install'
#
# Install Pmodules files.
#
build::install() {
while (( $# > 0 )); do
case $1 in
--debug )
set -x
;;
--prefix )
prefix="$2"
shift 1
;;
--prefix=* )
prefix="${1#*=}"
;;
-f | --force )
opt_force='yes'
--install_root | --install_root=* )
if [[ $1 == *=* ]]; then
INSTALL_ROOT="${1#*=}"
else
INSTALL_ROOT="$2"
shift 1
fi
;;
--help | -h | -\? )
pmodules::help_install
build::help_install
;;
-* )
std::die 1 "$1: illegal option"
illegal_option 'install' "$1"
;;
* )
std::die 1 "$1: illegal argument to sub-command 'install'."
illegal_arg 'install' "$1"
;;
esac
shift 1
done
local config_file="${prefix}/${CONFIG_DIR}/${CONFIG_FILE}"
read_config_file 'config_file'
: ${INSTALL_ROOT:=${DEFAULT_INSTALL_ROOT}}
PREFIX="${INSTALL_ROOT}/${PMOD_DIR}"
read_config_file "${INSTALL_ROOT}/${CONFIG_FILE}"
###
#
# begin installation
#
echo "Configuration:"
echo " root of Pmodules environment: ${PMODULES_ROOT}"
echo " Pmodule prefix: ${PMODULES_HOME}"
sed_cmd="s:@PMODULES_HOME@:${PMODULES_HOME}:g;"
sed_cmd+="s:@PMODULES_VERSION@:${PMODULES_VERSION}:g;"
sed_cmd+="s:@MODULES_VERSION@:${MODULES_VERSION}:g;"
sed_cmd+="s:@PMODULES_DISTFILESDIR@:${PMODULES_DISTFILESDIR}:g;"
sed_cmd+="s:@PMODULES_TMPDIR@:${PMODULES_TMPDIR}:g;"
sed_cmd+="s:@TCLSHDIR@:${PMODULES_HOME}/${UTILBIN_DIR}:g;"
sed_cmd+="s:@pager@::g;"
sed_cmd+="s:@pageropts@::g;"
sed_cmd+="s:@etcdir@:${PMODULES_ROOT}/${CONFIG_DIR}:g;"
echo " root of Pmodules environment: ${INSTALL_ROOT}"
echo " Pmodule prefix: ${PREFIX}"
sed_cmd+="s:@PMODULES_VERSION@:${VERSION}:g;"
sed_cmd+="s:@VERSIONING@:#:g;"
sed_cmd+="s:@prefix@:${PMODULES_HOME}:g;"
sed_cmd+="s:@initdir@:${PMODULES_HOME}/init:g;"
sed_cmd+="s:@MODULES_RELEASE@:${PMODULES_VERSION}:g;"
sed_cmd+="s:@BASH@:${PMODULES_HOME}/${UTILBIN_DIR}/bash:g;"
sed_cmd+="s:@MODULECMD@:${PMODULES_HOME}/${UTILBIN_DIR}/modulecmd.bash:g;"
sed_cmd+="s:@MODMANAGE@:${PMODULES_HOME}/${UTILBIN_DIR}/modmanage.bash:g;"
sed_cmd+="s:@BASH@:${PREFIX}/${UTILBIN_DIR}/bash:g;"
sed_cmd+="s:@MODULECMD@:${PREFIX}/${UTILBIN_DIR}/modulecmd.bash:g;"
sed "${sed_cmd}" "${SRC_DIR}/profile.bash.in" \
> "${INSTALL_ROOT}/${CONFIG_DIR}/profile.bash-${VERSION}"
sed "${sed_cmd}" "${SRC_DIR}/profile.csh.in" \
> "${INSTALL_ROOT}/${CONFIG_DIR}/profile.csh-${VERSION}"
sed "${sed_cmd}" "${SRC_DIR}/profile.zsh.in" \
> "${INSTALL_ROOT}/${CONFIG_DIR}/profile.zsh-${VERSION}"
chmod 0644 "${INSTALL_ROOT}/${CONFIG_DIR}"/*-${VERSION}
test -e "${INSTALL_ROOT}/${CONFIG_DIR}/profile.bash" || \
install -m 0644 "$_-${VERSION}" "$_"
test -e "${INSTALL_ROOT}/${CONFIG_DIR}/profile.csh" || \
install -m 0644 "$_-${VERSION}" "$_"
test -e "${INSTALL_ROOT}/${CONFIG_DIR}/profile.zsh" || \
install -m 0644 "$_-${VERSION}" "$_"
sed "${sed_cmd}" "${SRC_DIR}/modulecmd.in" \
> "${PREFIX}/bin/modulecmd"
chmod 0755 "${PREFIX}/bin/modulecmd"
sed "${sed_cmd}" "${SRC_DIR}/modulecmd.bash.in" \
> "${PREFIX}/libexec/modulecmd.bash"
chmod 0755 "${PREFIX}/libexec/modulecmd.bash"
sed "${sed_cmd}" "${SRC_DIR}/modulecmd.tcl.in" \
> "${PREFIX}/libexec/modulecmd.tcl"
chmod 0755 "${PREFIX}/libexec/modulecmd.tcl"
sed "${sed_cmd}" "${SRC_DIR}/libpmodules.bash.in" \
> "${PREFIX}/lib/libpmodules.bash"
chmod 0755 "${PREFIX}/lib/libpmodules.bash"
sed "${sed_cmd}" "${SRC_DIR}/modbuild.in" \
> "${PREFIX}/bin/modbuild"
chmod 0755 "${PREFIX}/bin/modbuild"
sed "${sed_cmd}" "${SRC_DIR}/modmanage.in" \
> "${PREFIX}/bin/modmanage"
chmod 0755 "${PREFIX}/bin/modmanage"
sed "${sed_cmd}" "${SRC_DIR}/modmanage.bash.in" \
> "${PREFIX}/libexec/modmanage.bash"
chmod 0755 "${PREFIX}/libexec/modmanage.bash"
test -e "${INSTALL_ROOT}/${CONFIG_FILE}" || \
install -m 0644 "$_" "${INSTALL_ROOT}/${CONFIG_DIR}"
sed "${sed_cmd}" "${SRC_DIR}/profile.bash.in" > "${PMODULES_ROOT}/${CONFIG_DIR}/profile.bash-${PMODULES_VERSION}"
sed "${sed_cmd}" "${SRC_DIR}/profile.csh.in" > "${PMODULES_ROOT}/${CONFIG_DIR}/profile.csh-${PMODULES_VERSION}"
sed "${sed_cmd}" "${SRC_DIR}/profile.zsh.in" > "${PMODULES_ROOT}/${CONFIG_DIR}/profile.zsh-${PMODULES_VERSION}"
chmod 0644 "${PMODULES_ROOT}/${CONFIG_DIR}"/*-${PMODULES_VERSION}
install -m 0755 "${SRC_DIR}/yq.$(uname -m)_$(uname -s)" "${PREFIX}/libexec/yq"
install -m 0644 "${SRC_DIR}/bash" "${PREFIX}/init"
install -m 0644 "${SRC_DIR}/bash_completion" "${PREFIX}/init"
install -m 0644 "${SRC_DIR}/csh" "${PREFIX}/init"
install -m 0644 "${SRC_DIR}/zsh" "${PREFIX}/init"
test -e "${PMODULES_ROOT}/${CONFIG_DIR}/profile.bash" || \
install -m 0644 "$_-${PMODULES_VERSION}" "$_"
test -e "${PMODULES_ROOT}/${CONFIG_DIR}/profile.csh" || \
install -m 0644 "$_-${PMODULES_VERSION}" "$_"
test -e "${PMODULES_ROOT}/${CONFIG_DIR}/profile.zsh" || \
install -m 0644 "$_-${PMODULES_VERSION}" "$_"
sed "${sed_cmd}" "${SRC_DIR}/modulecmd.in" > "${PMODULES_HOME}/bin/modulecmd"
chmod 0755 "${PMODULES_HOME}/bin/modulecmd"
sed "${sed_cmd}" "${SRC_DIR}/modulecmd.bash.in" > "${PMODULES_HOME}/libexec/modulecmd.bash"
chmod 0755 "${PMODULES_HOME}/libexec/modulecmd.bash"
sed "${sed_cmd}" "${SRC_DIR}/modulecmd.tcl.in" > "${PMODULES_HOME}/libexec/modulecmd.tcl"
chmod 0755 "${PMODULES_HOME}/libexec/modulecmd.tcl"
sed "${sed_cmd}" "${SRC_DIR}/libpmodules.bash.in" > "${PMODULES_HOME}/lib/libpmodules.bash"
chmod 0755 "${PMODULES_HOME}/lib/libpmodules.bash"
sed "${sed_cmd}" "${SRC_DIR}/modbuild.in" > "${PMODULES_HOME}/bin/modbuild"
chmod 0755 "${PMODULES_HOME}/bin/modbuild"
sed "${sed_cmd}" "${SRC_DIR}/modmanage.in" > "${PMODULES_HOME}/bin/modmanage"
chmod 0755 "${PMODULES_HOME}/bin/modmanage"
sed "${sed_cmd}" "${SRC_DIR}/modmanage.bash.in" > "${PMODULES_HOME}/libexec/modmanage.bash"
chmod 0755 "${PMODULES_HOME}/libexec/modmanage.bash"
test -e "${PMODULES_ROOT}/${CONFIG_DIR}/Pmodules.conf" || \
install -m 0644 "${SRC_DIR}/Pmodules.conf" "${PMODULES_ROOT}/${CONFIG_DIR}"
install -m 0644 "${SRC_DIR}/bash" "${PMODULES_HOME}/init"
install -m 0644 "${SRC_DIR}/bash_completion" "${PMODULES_HOME}/init"
install -m 0644 "${SRC_DIR}/csh" "${PMODULES_HOME}/init"
install -m 0644 "${SRC_DIR}/zsh" "${PMODULES_HOME}/init"
install -m 0644 "${SRC_DIR}/libpbuild.bash" "${PMODULES_HOME}/lib"
install -m 0644 "${SRC_DIR}/libpbuild_dyn.bash" "${PMODULES_HOME}/lib"
install -m 0644 "${SRC_DIR}/libstd.bash" "${PMODULES_HOME}/lib"
install -m 0755 -d "${PMODULES_HOME}/lib/Pmodules"
install -m 0644 "${SRC_DIR}/libmodules.tcl" "${PMODULES_HOME}/lib/Pmodules"
install -m 0644 "${SRC_DIR}/libpbuild.bash" "${PREFIX}/lib"
install -m 0644 "${SRC_DIR}/libstd.bash" "${PREFIX}/lib"
install -m 0755 -d "${PREFIX}/lib/Pmodules"
install -m 0644 "${SRC_DIR}/libmodules.tcl" "${PREFIX}/lib/Pmodules"
{
PATH="${PMODULES_HOME}/${UTILBIN_DIR}:${PATH}"
cd "${PMODULES_HOME}/lib/Pmodules"
PATH="${PREFIX}/${UTILBIN_DIR}:${PATH}"
cd "${PREFIX}/lib/Pmodules"
"${BOOTSTRAP_DIR}/mkindex.tcl"
}
install -m 0755 -d "${PMODULES_ROOT}/Tools/modulefiles/Pmodules"
install -m 0644 "${SRC_DIR}/modulefile" "${PMODULES_ROOT}/Tools/modulefiles/Pmodules/${PMODULES_VERSION}"
mkdir -p "${PMODULES_ROOT}/Libraries/modulefiles"
install -m 0644 \
"${SRC_DIR}/modulefile" \
"${INSTALL_ROOT}/Tools/modulefiles/Pmodules/${VERSION}"
echo "Done..."
}
#=============================================================================
#
declare -a build_opts=()
build_opts+=( '--bootstrap' )
declare subcmd=''
declare -a subcmd_args=()
@@ -552,6 +523,7 @@ while (( $# > 0 )); do
shift 1
subcmd_args=( "$@" )
shift $#
break
;;
* )
std::die 1 "Invalid sub-command '$1'.\n\nUse 'build --help' to get help..."
@@ -562,5 +534,10 @@ done
[[ -n "${subcmd}" ]] || std::die 1 "Missing sub-command.\n\nUse 'build --help' to get help..."
pmodules::${subcmd} "${subcmd_args[@]}"
build::${subcmd} "${subcmd_args[@]}"
# Local Variables:
# mode: sh
# sh-basic-offset: 8
# tab-width: 8
# End:
+9
View File
@@ -0,0 +1,9 @@
DefaultGroups: Tools:Programming
DefaultReleaseStages: stable
ReleaseStages: unstable:stable:deprecated
TmpDir: @PMODULES_TMPDIR@
DistfilesDir: @PMODULES_DISTFILESDIR@
Overlays:
base:
install_root: @INSTALL_ROOT@
-18
View File
@@ -1,18 +0,0 @@
#!/bin/bash
#
: ${PMODULES_ROOT:=@PMODULES_ROOT@}
: ${PMODULES_DISTFILESDIR:=@PMODULES_DISTFILESDIR@}
: ${PMODULES_TMPDIR:=@PMODULES_TMPDIR@}
export PMODULES_ROOT
export PMODULES_DISTFILESDIR
export PMODULES_TMPDIR
declare -x PMODULES_CONFIG_DIR='config'
declare -x PMODULES_MODULEFILES_DIR='modulefiles'
declare -x PMODULES_HOME="${PMODULES_ROOT}/Tools/Pmodules/${PMODULES_VERSION}"
declare -x DefaultGroups='Tools Programming'
declare ReleaseStages=':unstable:stable:deprecated:'
declare DefaultReleaseStages='stable'
+276
View File
@@ -0,0 +1,276 @@
= Overlays
:TOC:
:sectnums:
Please note: This document is work in progress!
== What are Overlays and for what can they be used?
== Overlay configuration
=== System wide
==== Example
.A system wide configuration file `/opt/psi/config/Pmodules.yaml`
====
....
DefaultGroups: Tools:Programming
DefaultReleaseStages: stable
ReleaseStages: unstable:stable:deprecated
TmpDir: /opt/psi/var/tmp/${USER}
DistfilesDir: /opt/psi/var/distfiles
Overlays:
base:
install_root: /opt/psi
modulefiles_root: /opt/psi
devel:
install_root: /opt/psi
modulefiles_root: ${HOME}/modulefiles
....
====
==== YAML Format
.Format
====
....
Overlays:
<name>:
install_root: <installation-root>
modulefiles_root: <modulefiles-root>
type: <type of overlay>
...
....
====
`<name>`:: Name of overlay. Note: whitespace in the name is not supported.
`<name>.install_root`:: The root of the software installation. This key is mandatory.
`<name>.modulefiles_root`:: The root of the modulefile hierarchy. This key is optional and defaults to `<name>.install_root>
`<name>.type`:: The type of the overlay, see below. This key is optional and the default value is `n`.
==== Overlay types
`n`:: Normal overlay.
`h`:: Hiding overlay.
`r`:: Replacing overlay.
=== User defined overlays
Each user can define his own overlays in `$HOME/.Pmodules/Pmodules.yaml`.
== Working with overlays
== Building modules
The old format of the variants file is simple but very limited and almost impossible to extend for new features. To overcome the limitations a new format using YAML for variants files has been introduced. For the time being both format are supported. But it is highly recommended to use the YAML format for new modules and to migrate existing variants files in the old format to the new.
=== With a YAML variants file
Information is in the YAML configuration file:
==== General structure
name:: The name of the Pmodule without version. Example: `hdf5`
group:: group the module is in. Example: `MPI`
overlay:: (optional) overlay, defaults to `base` if not specified
relstage:: (optional) default release stage. If not defined,
it defaults to `unstable`,
url:: (optional) download URL, Must be either define here or per
version. The environment variables `$P`, `$V`, `$V_PKG` etc. can be used. The downloaded file is stored with the file name `$P-${V_PKG}.ext`. Whereby `ext` is derived from the URL.
addon_sources:: (optional) array of sources for add-ons. Example: `git-lfs`
use:: (optional) use flags.
systems:: (optional) array of supported systems, defaults to any. Example: `[rhel6 rhel7]`.
compilers:: (optional) array pf supported compilers, defaults to any. Example: `[gcc intel]`.
compile-in-src:: (optional) compile in source tree, allowed vaules are `yes` and `no`, the default is `no`.
configure-with:: (optional) If a software supports CMake and autotools, this specifies the tool to use. Allowed values are `autotools`, `cmake`, `auto`, `none`. The default is `auto`.
versions:: array of `version` objects
==== Objects of type `addon_source`
name:: unique name
url:: download URL.
shasum:: (optional) SHA256 sum.
==== Objects of type `version`
dependencies:: other run-time dependencies, if any
build_with:: dependencies required for building, if any
overlay:: (optional) override the default. Example: `devel`.
relstage:: (optional) override the default. Example: `unstable`.
addon_sources:: (optional) array of sources for add-ons.
systems:: (optional) override the default
compilers:: (optional) override the default
compile-in-src:: (optional) override the default
configure-with:: (optional) override the default
variants:: array of `variant` objects
==== Object of type `variant`
with:: hierarchical dependencies, `with` must be specified only
for hierarchical groups.
dependencies:: other run-time dependencies, if any
build_requires:: dependencies required for building, if any
addon_sources:: (optional) array of sources for add-ons.
use:: (optional) use flags
overlay:: (optional) override the default. Example: `devel`.
relstage:: (optional) override the default. Example: `unstable`.
systems:: (optional) override the default
compilers:: (optional) override the default
==== Example of a configuration files in YAML format
.YAML configuration file for serial HDF5
====
....
name: hdf5
group: MPI
overlay: base
relstage: stable
url: https://support.hdfgroup.org/ftp/HDF5/releases/$P-${V_MAJOR}.${V_MINOR}/$P-${V_PKG}/src/$P-${V_PKG}.tar.gz
systems: [rhel6,rhel7,rhel8]
compilers: [gcc,intel,pgi]
compile_in_sourcetree: no
configure_with: auto
versions:
1.12.2:
shasum: 3016ea56a175d2ca7f2568c8016420f7a2aad8f95e214fe7fa5485f4b80fbe51
variants:
- group_deps:
compiler:
gcc: [5.5.0,6.5.0,7.5.0,10.2.0,10.3.0]
intel: ....
mpi:
openmpi: [...]
#hdf5:
runtime_deps:
cuda:10.0
...
build_deps:
name:version
- with:
- gcc: [8.5.0,9.5.0,11.3.0,12.1.0]
relstage: unstable
overlay: devel
....
====
.YAML configuration file for HDF5 (serial and parallel)
====
....
name: hdf5
overlay: base
relstage: stable
url: https://support.hdfgroup.org/ftp/HDF5/releases/$P-${V_MAJOR}.${V_MINOR}/$P-${V_PKG}/src/$P-${V_PKG}.tar.gz
systems: [rhel6 rhel7 rhel8]
compilers: [gcc intel pgi]
compile_in_sourcetree: no
configure_with: auto
versions:
1.12.2:
shasum: 3016ea56a175d2ca7f2568c8016420f7a2aad8f95e214fe7fa5485f4b80fbe51
variants:
- with:
- gcc [5.5.0 6.5.0 7.5.0 10.2.0 10.3.0]
group: Compiler
- with:
- gcc [8.5.0,9.5.0,11.3.0,12.1.0]
- openmpi [4.0.5-2]
group: MPI
relstage: unstable
overlay: devel
....
====
.YAML configuration file for Git with git-lfs
====
....
name: git
defaults:
group: Tools
overlay: base
relstage: stable
url: https://mirrors.edge.kernel.org/pub/software/scm/$P/$P-${V_PKG}.tar.xz
systems: [rhel6 rhel7 rhel8]
compilers: [gcc]
compile_in_sourcetree: yes
configure_with: auto
script: build
addons:
git-lfs/3.2.0:
- url: https://github.com/git-lfs/git-lfs/archive/refs/tags/v3.2.0.tar.gz
shasum: f8e6bbe043b97db8a5c16da7289e149a3fed9f4d4f11cffcc6e517c7870cd9e5
build_requires: [go/1.19]
build_script: build-git-lfs
versions:
2.37.2:
shasum: 4c428908e3a2dca4174df6ef49acc995a4fdb1b45205a2c79794487a33bc06e5
variants:
- dependencies: TclTk/8.6.9
build_requires: [perl/5.30.0 asciidoc/8.6.9-1 xmlto/0.0.28 gettext/0.19.8]
relstage: unstable
overlay: devel
addons:
git-lfs: 3.2.0
....
====
==== Format specification
.YAML format
....
name: <Pmodule-name-without-version>
source_url: "https://..."
overlay: <opt-default-overlay>
group: <default-group>
relstage: <opt-default-release-stage>
systems: [opt-array-of-supported]
compilers: [opt-array-of-supported-compilers]
compile-in-src: yes|no
configure-with: cmake|autotools|auto|none
versions:
- <version1>:
overlay: <overlay>
group: <group>
relstage: <release-stage>
systems: [supported-systems]
sources:
- url: <url>
shasum: <shasum>
compilers: <default-list-of-supported-compilers>
compile-in-src: yes|no
configure-with: cmake|autotools|auto|none
variants:
- with:
- gcc: [5.5.0,6.5.0,7.5.0,10.2.0,10.3.0]
dependencies: [array-of-runtime-deps]
build_requires: [array-of-build-deps]
relstage: <release-stage>
use: [array-of-use-flags]
overlay: <overlay>
...
....
==== Defaults
Default values can be overriden per version/variant.
`overlay`:: The default overlay the module will be installed in. This value can be overriden for dedicated versions/variants.
`systems`:: The default for supported systems.
==== Versions and Variants
`<name>/<version>`:: An array with variants for this version.
`<name>/<version>.[i].with`:: Hierarchical dependencies for variant `i`.
`<name>/<version>.[i].dependencies`:: Build/run-time dependencies for variant `i`.
`<name>/<version>.[i].relstage`:: Relase stage of variant `i`.
`<name>/<version>.[i].overlay`:: Overlay of variant `i`.
`<name>/<version>.[i].systems`:: Supported systems.
=== Legacy format
+42
View File
@@ -0,0 +1,42 @@
#!/bin/bash
#
# https://www.gnu.org/software/bash/
#
P=bash
V=${BASH5_VERSION:-5.1.16}
FNAME="$P-$V.tar.gz"
DOWNLOAD_URL="https://ftp.gnu.org/gnu/$P/${FNAME}"
source "$(dirname "$0")/librecipes.bash"
#---
# configure
mkdir -p "${BUILD_DIR}" && cd "$_" || exit ${PB_ERR_SYSTEM}
loadablesdir="${PREFIX}/${UTILBIN_DIR}/builtins" \
"${SRC_DIR}/configure" \
--prefix="${PREFIX}" \
--bindir="${PREFIX}/${UTILBIN_DIR}" \
|| exit ${PB_ERR_CONFIGURE}
#---
# compile
make -j ${NJOBS} || exit ${PB_ERR_MAKE}
make -C examples/loadables -j ${NJOBS} || exit ${PB_ERR_MAKE}
#---
# install
make install || exit ${PB_ERR_INSTALL}
#---
# post-install
rm -rf "${PREFIX}/include/bash"
rm -rf "${PREFIX}/share/locale"
rm -rf "${PREFIX}/share/doc"
rm -rf "${PREFIX}/share/info"
rm -rf "${PREFIX}/share/man/man1/bash"*
#---
# Local Variables:
# mode: shell-script-mode
# sh-basic-offset: 8
# End:
+60
View File
@@ -0,0 +1,60 @@
#!/bin/bash
#
# https://www.tcl.tk
#
P=tcl
V=${TCL_VERSION:-8.6.12}
FNAME="$P$V-src.tar.gz"
DOWNLOAD_URL="https://prdownloads.sourceforge.net/tcl/${FNAME}"
source "$(dirname "$0")/librecipes.bash"
#---
# download
test -r "${SRC_FILE}" || curl -L --output "$_" "${DOWNLOAD_URL}" || exit ${PB_ERR_DOWNLOAD}
#---
# unpack
mkdir -p "${SRC_DIR}" && cd "$_" || exit ${PB_ERR_SYSTEM}
tar --directory "${SRC_DIR}" --strip-components 1 -xv -f "${SRC_FILE}" || exit ${PB_ERR_UNTAR}
#---
# configure
mkdir -p "${BUILD_DIR}" && cd "$_" || exit ${PB_ERR_SYSTEM}
case $(uname -s) in
Linux )
srcdir="${SRC_DIR}/unix"
;;
Darwin )
srcdir="${SRC_DIR}/macosx"
;;
* )
echo "Oops: unsupported OS!" 1>&2
exit ${PB_ERR_SYSTEM}
;;
esac
"${srcdir}"/configure \
--prefix="${PREFIX}" \
--bindir="${PREFIX}/${UTILBIN_DIR}" \
--mandir="${PREFIX}/share/man" \
--enable-shared=no \
|| exit ${PB_ERR_CONFIGURE}
#---
# compile & install
make -j ${NJOBS} || exit ${PB_ERR_MAKE}
make install || exit ${PB_ERR_INSTALL}
#---
# post-install
{ cd "${PREFIX}/${UTILBIN_DIR}" && rm -f tclsh && ln -fs tclsh${V%.*} tclsh; };
rm -rf "${PREFIX}/share/man/man1/tclsh.1"
rm -rf "${PREFIX}/share/man/man3"
rm -rf "${PREFIX}/share/man/mann"
#---
# Local Variables:
# mode: shell-script-mode
# sh-basic-offset: 8
# End:
+48
View File
@@ -0,0 +1,48 @@
#!/bin/bash
#
# https://core.tcl-lang.org
#
P=tcllib
V=${TCLLIB_VERSION:-1.21}
FNAME="$P-$V.tar.gz"
DOWNLOAD_URL="https://core.tcl-lang.org/tcllib/uv/${FNAME}"
source "$(dirname "$0")/librecipes.bash"
#---
# download
test -r "${SRC_FILE}" || curl -L --output "$_" "${DOWNLOAD_URL}" || exit ${PB_ERR_DOWNLOAD}
#---
# unpack
mkdir -p "${SRC_DIR}" && cd "$_" || exit ${PB_ERR_SYSTEM}
tar --directory "${SRC_DIR}" --strip-components 1 -xv -f "${SRC_FILE}" || exit ${PB_ERR_UNTAR}
# configure
mkdir -p "${BUILD_DIR}" && cd "$_" || exit ${PB_ERR_SYSTEM}
"${SRC_DIR}"/configure \
--prefix="${PREFIX}" \
--mandir="${PREFIX}/share/man" \
|| exit ${PB_ERR_CONFIGURE}
#---
# compile & install
make -j ${NJOBS} || exit ${PB_ERR_MAKE}
make install || exit ${PB_ERR_INSTALL}
#---
# post-install
rm -rf "${PREFIX}/share/man/mann"
rm -f "${PREFIX}/bin/dtplite"
rm -f "${PREFIX}/bin/mkdoc"
rm -f "${PREFIX}/bin/nns"
rm -f "${PREFIX}/bin/nnsd"
rm -f "${PREFIX}/bin/nnslog"
rm -f "${PREFIX}/bin/page"
rm -f "${PREFIX}/bin/pt"
rm -f "${PREFIX}/bin/tcldocstrip"
#---
# Local Variables:
# mode: shell-script-mode
# sh-basic-offset: 8
# End:
+59
View File
@@ -0,0 +1,59 @@
#!/bin/bash
#
# https://core.tcl-lang.org
#
P=modules
V=${MODULES_VERSION:-3.2.10.1}
FNAME="$P-$V.tar.gz"
DOWNLOAD_URL="https://amas.web.psi.ch/Downloads/$P/$P-$V.tar.bz2"
source "$(dirname "$0")/librecipes.bash"
#---
# configure
mkdir -p "${BUILD_DIR}" && cd "$_" || exit ${PB_ERR_SYSTEM}
case $(uname -s) in
Linux )
declare -x LIBS="-lz -lpthread"
;;
Darwin )
declare -x LIBS="-lz -framework CoreFoundation"
;;
* )
echo "Oops: unsupported OS!" 1>&2
exit ${PB_ERR_SYSTEM}
;;
esac
CPPFLAGS="-DUSE_INTERP_ERRORLINE" \
"${SRC_DIR}"/configure \
--prefix="${PREFIX}" \
--exec-prefix="${PREFIX}" \
--with-module-path="${PREFIX%%/Tools*}/Tools/${PMODULES_MODULEFILES_DIR}" \
--with-tcl="${PREFIX}/lib" \
--without-x \
--disable-versioning \
|| exit ${PB_ERR_CONFIGURE}
#---
# compile & install
make -j ${NJOBS} || exit ${PB_ERR_MAKE}
make install || exit ${PB_ERR_INSTALL}
#---
# post-install
mkdir -p "${PREFIX}/share/man/man1"
mkdir -p "${PREFIX}/share/man/man4"
mv -v "${PREFIX}/Modules/share/man/man1/module.1" "${PREFIX}/share/man/man1"
mv -v "${PREFIX}/Modules/share/man/man4/modulefile.4" "${PREFIX}/share/man/man4"
mkdir -p "${PREFIX}/libexec"
cp -v "${BUILD_DIR}/modulecmd" "${PREFIX}/libexec/modulecmd.bin" || exit 1
rm -rf "${PREFIX}/Modules"
rm -f "${PREIX}"/init/{ksh,perl.pm,python.py,ruby.rb,cmake,.modulespath}
#---
# Local Variables:
# mode: sh
# sh-basic-offset: 8
# tab-width: 8
# End:
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env bash
#
# https://frodo.looijaard.name/project/getopt
#
P=getopt
V=${GETOPT_VERSION:-1.1.6}
FNAME="$P-$V.tar.gz"
DOWNLOAD_URL="http://frodo.looijaard.name/system/files/software/$P/$P-$V.tar.gz"
#---
# build on macOS only
[[ $(uname -s) == 'Darwin' ]] || exit 0
#---
source "$(dirname "$0")/librecipes.bash"
#---
# configure
# nothing to configure but we need gettext from Macports
if [[ ! -d '/opt/local/bin' ]] || [[ ! -x '/opt/local/bin/msgfmt' ]]; then
echo "gettext port from Macports is required to build 'getopt'!" 1>&2
exit 1
fi
#---
# compile
PATH+=':/opt/local/bin'
declare -x C_INCLUDE_PATH="${PREFIX}/include:/opt/local/include"
declare -x LDFLAGS="/opt/local/lib/libintl.a /opt/local/lib/libiconv.a -framework CoreFoundation"
declare -x LIBRARY_PATH="${PREFIX}/lib"
make -e all || exit 1
#---
# install
declare -x DESTDIR="${PREFIX}"
declare -x prefix=''
#PATH="${PREFIX}/${UTILBIN_DIR}:${PATH}"
make -e install
#---
# post-install
mv "${PREFIX}/bin/getopt" "${PREFIX}/${UTILBIN_DIR}"
rm -rf "${PREFIX}/man"
rm -rf "${PREFIX}/share/locale"
#---
# Local Variables:
# mode: sh
# sh-basic-offset: 8
# tab-width: 8
# End:
+43
View File
@@ -0,0 +1,43 @@
#!/bin/bash
#
# https://www.gnu.org/software/findutils/
#
P=findutils
V=${FINDUTILS_VERSION:-4.9.0}
FNAME="$P-$V.tar.xz"
DOWNLOAD_URL="https://ftp.gnu.org/gnu/$P/${FNAME}"
#---
# build on macOS only
[[ $(uname -s) == 'Darwin' ]] || exit 0
#---
source "$(dirname "$0")/librecipes.bash"
#---
# configure
mkdir -p "${BUILD_DIR}" && cd "$_" || exit ${PB_ERR_SYSTEM}
loadablesdir="${PREFIX}/${UTILBIN_DIR}/builtins" \
"${SRC_DIR}/configure" \
--prefix="${PREFIX}" \
--bindir="${PREFIX}/${UTILBIN_DIR}" \
|| exit ${PB_ERR_CONFIGURE}
#---
# compile & install
make -j ${NJOBS} || exit ${PB_ERR_MAKE}
make install || exit ${PB_ERR_INSTALL}
#---
# post-install
rm -vf "${PREFIX}/share/man/man5/locatedb.5"
rm -vf "${PREFIX}/share/man/man1/updatedb.1"
rm -vf "${PREFIX}/share/man/man1/xargs.1"
rm -vf "${PREFIX}/share/man/man1/locate.1"
rm -vf "${PREFIX}/share/man/man1/find.1"
#---
# Local Variables:
# mode: shell-script-mode
# sh-basic-offset: 8
# End:
+91
View File
@@ -0,0 +1,91 @@
#!/bin/bash
set -x
set -o errexit
set -o pipefail
shopt -s nullglob
if (( $# == 0 )); then
echo "Usage: $0 <dest-dir>" 1>&2
exit 1
fi
PREFIX="$1"
if [[ ! -d ${PREFIX} ]]; then
echo "Destinstion directory '${PREFIX}' does not exist! Aborting..." 1>&2
exit 2
fi
TMP_DIR="${PMODULES_TMPDIR:-/var/tmp/${USER}}"
DOWNLOADS_DIR="${PMODULES_DISTFILESDIR:-${TMP_DIR}/Downloads}"
SRC_DIR="${TMP_DIR}/$P-$V/src"
BUILD_DIR="${TMP_DIR}/$P-$V/build"
SRC_FILE="${DOWNLOADS_DIR}/${FNAME}"
declare -ix PB_ERR_ARG=1
declare -ix PB_ERR_SETUP=2
declare -ix PB_ERR_SYSTEM=3
declare -ix PB_ERR_DOWNLOAD=4
declare -ix PB_ERR_UNTAR=5
declare -ix PB_ERR_CONFIGURE=6
declare -ix PB_ERR_MAKE=7
declare -ix PB_ERR_PRE_INSTALL=8
declare -ix PB_ERR_INSTALL=9
declare -ix PB_ERR_POST_INSTALL=10
declare -ix PB_ERR=255
declare -ix NJOBS=4
pb_exit() {
local -i ec=$?
if [[ -n "${BASH_VERSION}" ]]; then
local -i n=${#BASH_SOURCE[@]}
local -r recipe_name="${BASH_SOURCE[n]}"
else
local -r recipe_name="${ZSH_ARGZERO}"
fi
echo -n "${recipe_name}: "
if (( ec == 0 )); then
echo "done!"
elif (( ec == PB_ERR_ARG )); then
echo "argument error!"
elif (( ec == PB_ERR_SETUP )); then
echo "error in setting everything up!"
elif (( ec == PB_ERR_SYSTEM )); then
echo "unexpected system error!"
elif (( ec == PB_ERR_DOWNLOAD )); then
echo "error in downloading the source file!"
elif (( ec == PB_ERR_UNTAR )); then
echo "error in un-taring the source file!"
elif (( ec == PB_ERR_CONFIGURE )); then
echo "error in configuring the software!"
elif (( ec == PB_ERR_MAKE )); then
echo "error in compiling the software!"
elif (( ec == PB_ERR_PRE_INSTALL )); then
echo "error in pre-installing the software!"
elif (( ec == PB_ERR_INSTALL )); then
echo "error in installing the software!"
elif (( ec == PB_ERR_POST_INSTALL )); then
echo "error in post-installing the software!"
else
echo "oops, unknown error!!!"
fi
exit ${ec}
}
#export -f pb_exit > /dev/null
trap "pb_exit" EXIT
#---
# download
mkdir -p "${DOWNLOADS_DIR}" || exit ${PB_ERR_SYSTEM}
test -r "${SRC_FILE}" || curl -L --output "$_" "${DOWNLOAD_URL}" || exit ${PB_ERR_DOWNLOAD}
#---
# unpack
mkdir -p "${SRC_DIR}" && cd "$_" || exit ${PB_ERR_SYSTEM}
tar --directory "${SRC_DIR}" --strip-components 1 -xv -f "${SRC_FILE}" || exit ${PB_ERR_UNTAR}
#---
# Local Variables:
# mode: shell-script-mode
# sh-basic-offset: 8
# End: