Pmodules/CHANGELOG.md

# Changelog of Pmodules

## Version 2.0.4

### modulecmd
* Bugfix in unloading modules. A versioned module couldn't be
  unloaded if only the name was specified.
  (#1364)
* Use of unset variable fixed in find_modulefile().
  (#1358)
* When loading multiple modules, `LD_LIBRARY_PATH` may not have
  been set correctly.
  (#1354)

### build system
* Don't set CC and CXX in pbuild::compile()
  (#1360)
* Don't pass an empty string as argument to configure/cmake.
  (#1356)
  
### other changesg
* Bash updated to version 5.3.9
  (#1366)
* Download URL for Modules fixed.
  (#1362)

## Version 2.0.3

### modulecmd
* Issue with Lmod spider command fixed.
  (#1348)
* Don't allow absolute path names for modules in an overlay.
  (#1344, #1350)
* It was not possible to load a module with it's absolute path.
  (#1343)
* Bugfix in unloading modules. Under certain conditions unloading a module
  failed due to a wrong check in the test whether a module is loaded or not.
  (#1340)
* Misleading error message fixed if system configuration file is not readable.
  (#1316, #1331)

### build system
* calling std::die in pbuild::install_docfiles() fixed.
  (#1342)
* typo in error message in function pbuild::configure() fixed.
  (#1338)
* Substitute environment variables in TmpDir and DistFilesDir in Pmodules.yaml
  (#1333)
* Function to simplify building modules with pip3 added. 
  (#1329)
* pbuild::install_docfiles() is now obsolete. If called it exists with
  an error message.
  (#1327)
* default values for `PMODULES_TMPDIR` and `PMODULES_DISTFILESDIR` are now 
  more general.
  (#1325, #1331)
* set CC and CXX if not defined. This was a bug introduced in 2.0.2
  (#1322)

### building Pmodules
* path to Lua installation fixed in recipe to build luarocks
  (#1320)

### other changes
* shell profile for RPM added

## Version 2.0.2

Migragtion to GitHub:
Old Gitlab issue numbers and GitHub numbers do not match!

### modulecmd
* bugfix: type error fixed
  (#1314)
* bugfix: if lsb_release is not installed on a Linux system, `/etc/os-release`
  was sourced. This triggered an error message, since the variable VERSION was
  defined  as read-only in modulecmd. 
  (#1299)
* fix: reference to documentation in help text fixed

### build system
* enhancement: if verbose output is enabled, run make in verbose 
  mode.
  (#1312)

### building Pmodules
* bugfix in recipe to build luaposix: specify installation path.
  (#1306)
* bugfix in recipe to build 'modules': we have to run a distclean
  before configuring.
  (#1304)
* enhancement: cBash updated to version 5.3
  (#1302)


## Version 2.0.1
### modulecmd
* There were several issues with the stack of used overlays. This could
  result in certain modules being incorrectly not listed with the 'avail'
  and 'search/spider' sub-commands.
  (#440)
* Code cleanup
  (#437, #443)
* Output hints in sub-command help if module exist but is
  not available.
  (#434, #436)
* (Re-)build cache only if a modulefile has been added/changed.
  (#430)
* Call `modulecmd`, `lmod` and `spider` via a wrapper function.
  (#428)

### build-system
* use of undefined LOADEDMODULES fixed
  (#451)
* rename of pbuild::add\_configure\_args() reverted
  (#449)
* Print message with used overlays
  (#442)
* Work-around for bash(?) bug in calling make
  (#441)
* Check for loaded modules missed Lua modules.
  (#439)
* Setting configure args was broken. When building multiple variants the
  configure args were appended instead of (re-)set.
  (#438)
* A module with the same name/version can now be in multiple overlays.
  Till now this was only possible with the base overlay plus another
  overlay. The option `--cleanup-modulefiles` can be used to remove
  all modulefiles from all overlays but the configured.
  (#432)
* Get hostname via the system binary `hostname(1)`. On a cluster with
  Slurm the compute nodes might inherit the hostname from a login node.
  In this case the value `$HOSTNAME` is wrong.
  (#431)
* The key `source` can now be used as alias for `urls`.
  The structure of `urls` has been extended. It's now possible to
  define the directory to unpack a source and add patch files.
  (#429)

### both
* Unused variable in doc-string fixed
  (#447)
* Code review: local variables must be declared local. This most likely
  fixes some strange behavior/bugs.
  (#433, #436)
* Call system binaries via a function. In this function `LD_PRELOAD` and 
  `LD_LIBRARY_PATH` are unset.
  (#427)

### build/compiling Pmodules
* don't use mkindex.tcl to create tclIndex file for libmodules.tcl.
  (#450)
* Sub-command to create a tar-ball added. This tar-ball can be used to 
  to build a Pmodules RPM.
  (#445)
* Bugfix in compiling Lmod.
  (#426)

### Repo
* Unsed files removed.
  (#446)
## Version 2.0.0
### modulecmd
* building/removing sub-packages fixed
  (#423)
* Call system binaries via a shell function. Unset `LD_LIBRARY_PATH` and
  `LD_PRELOAD` before calling the binary.
  (#422)
* bugfix in `module search`: not all matching modules were listed.
  (#421, #424)
* don't print a message after loading a stable module.
  (#420)
* cache timeout increased to 4h.
  (#419)
* `module whatis|appropos` fixed.
  (#418)
* `module search|spider` fixed for hierachical Lmod modules.
  (#417)
* Initialize `MANPATH` if not already set.
  (#416)
* reverse sort output of `module avail|search` only if called without
  patternen. 
  (#415)
* bugfix in loading Lmod modules.
  (#414)
* more verbose deprecated message.
  (#413)
* `module avail|search` must be case-insensitive.
  (#410)
* Under certain conditions `module avail` showed modules which were
  hidden/shadowed be an overlay. This change fixed the issue for 
  non-hierarchical groups but not 100% for hierarchical groups. 
  (#408)
* re-initialize the Pmodules env after a `module purge`
  (#406, #409)
* Don't export the environment variables `OSrelease` and `SystemCPU`
  in `PMODULES_ENV`. This causes issues on inhomogenious clusters.
  (#405)OSrelease und SystemCPU
* bugfix in defining doc-string fixed 
  (#404)
* bugfix in using/un-using groups
  (#403, #407)

### build-system
* The check whether the main package has to be rebuild was wrong.
  (#412)
* code to set RPATH simplified and made more robust.
  (#411)


### both

## Version 1.1.23
### modulecmd
* bugfix in read the Pmodules config: overlay names may contain dots. The 
  overlay name must be quoted in the query.
  (#387)
* bugfix in std::append_path() and std::prepend_path()
  (#386)
* bugfix in std::remove_path(), the function didn't work for more then
  one directory.
* support for Lmods created by Spack
  (#383, #385, #388, #393, #396, #400)
* bugfix: switching between Pmodules version now works without unsetting
  PMODULES_ENV.
  (#381)
* The init are now relocatable and do not depend on the directory
  set in $PMODULES_HOME
  (#376)

### build-system
* order of checks changes whether a module variant must be built on this
  system. Order is now: kernel, cpu, system/os/hostname.
  (#391)
* checkk add that a build dependency has been successfully loaded.
  (#390)
* bugfix in setting up the environment before building a module.
  (#389)
* bugfix in reading group dependencies from the YAML config file.
  (#382)

### Both
* Tcl, Lua, Lmod, yq and BASH updated to current versions.
  (#392, #394, #395)
* code review/refactoring
  (#377, #378, #379, #380, #397, #398, #399, #401)

## Version 1.1.22

### modulecmd
* excludes in overlays are now supporting regex notation
  {#353}
* the dependencies of a module are now defined in the modulefiles
  directory. This way it is possbile to define different dependencies 
  per overlay.
  (#338)
* fixes in loading/unloading Lua modules
  (#331)
* Create overlay directories if the do not already exist. 
  Print error message if the directories cannot be created
  (#326)
* An overlay can now add additional directories to `MODULEPATH`
  (#321)
* Review/optimze code to find modulefiles
  (#318, #319, #328)
* Ignore relative directories in `MODULEPATH`
  (#316)
* Add alias `spider` to `search` sub-command
  (#313)
* Print help text if `modulecmd` is called without argument
  (#312)
* fixes/improvements in module search command. Handling of
  directories outside Pmodules hierarchy fixed.
  (#301, #311)
* Improved support for Lmod 
  (#302, #325, #327)
* code review, cleanup and bugfixes 
  (#299, #300, #314, #315, #317, #332, #333, #334, #337, #339, #346,
  #352, #360, #363, #364)

### build-system
* YAML anchors can now be used in the module config file.
  (#374)
* The modulefile to be installed can now be configured
  (#371)
* The function used to build a module can now be defined in the 
  YAML config file.
  (#359)
* Legacy build function removed
  (#355)
* Cleanup/review reading YAML config files
  (#348)
* Use regex instead of glob to match systems
  (#343)
* Type checks of YAML nodes added
  (#342)
* Configure required overlays in YAML config and load them before
  starting to build a module.
  (#341)
* `pbuild::add_configure_args()` isn't deprecated any more. Sometimes
  it is useful in the build-script
  (#322).
* `none` added as unpacker.
  (#307)
* Don't override existing RPATH. This is an important fix for packages
  like conda.
  (#304)
* Force rebuild if build script is called with `--clean-install`.
  (#305)
* code review, cleanup and bugfixes .
  (#299, #300, #303, #306, #310, #320, #323, #324, #335, #336, #345,
  #349, #350, #351, #354, #362, #362, #365, #368, #369, #370, #372)

### both
* The default Pmodules system config can now be overwritten with the 
  environment variables `PMODULES_CONFIG_FILE`.
  (#373)
* support for aarch64 added
* sles15 added to list of recognized Linux distributions
  (#309)
* bugfixes
  (#308, #367)

## Version 1.1.21
### modulecmd
* update bash to 5.2.21 and Tcl to 8.6.14 (#274)
* option '--debug' enables shell xtrace, option '--verbose'
  informational messages (#275)
* for modules not in the Pmodules hierarchy: display module
  name only, not full qualified name of the modulefile. (#285)
* perform module search in all directories in `MODULEPATH
  outside the Pmodules hierarchy. Till know a module search
  was limited to the Pmodules hierarchy (#286).
* bugfix: loading multiple modules with the same load command
  is working again (#281)
* more bugfixes (#282)

### build-system
* The tool '7z' can know be used to unpack an archive. The tool
  to unpack an archive can be configured in the YAML config file.
  For now `tar` and `7z` are supported. (#273, #278, #289)
* set `RPATH` in executables and shared libraries to 
  `${PMODULES_ROOT}/lib`. The set RPATH is relative using `$ORIGIN`.
  (#279, #280, #290)
* expand `${PREFIX}` while parsing the YAML configuration file (#287)
* the download directory can now be configured in the YAML
  configuration file (#293)
* support for different target CPU and OS kernels added (#294)
* more bugfixes (#277, #295)
* cleanup (#292)

## Version 1.1.20
### modulecmd
* Bugfix in sub-command 'unuse' with directory as argument.
  Due to a missing return statement, the sub-command exited 
  with an error.
  (issue #262)
* Bugfixes in the sub-commands to save/list/restore collections.
  (issue #263)


### build-system
* Support for sub-packages added.
  (issuesg #264, #266)
* Code cleanup.
  (issues #265, #269)
* Option '--overlay' is only allowed with legacy configuration
  files. Check added.
  (issue #267)
* Check mode for YAML configuration files added.
  (issue #268)

## Version 1.1.19
### modulecmd
* BUGFIX: parsing the version number and setting the variables
  V_MAJOR, V_MINOR, V_PATCHLVL was broken in cases where the
  version number consist of less then three numbers and plus a
  suffix.
  (issue #248)
* BUGFIX: never cut output of load hints. This could happen in
  VScode/VScodium. Don't cut output of 'module search' if not
  running in a terminal.
  (issues #250,#255)

### build-system
* Option '--clean-install' added. If this option is set, the 
  module is removed before building, if the module already
  exist.
  (issue #247)
* Arguments to CMake/autotools and required patches can now
  be defined in the YAML configuration file.
  (issues #249, #253, #256)
* Check of consinstency of group dependencies added.
  (issue #251)
* The number of directory components to be removed while
  un-taring can now be configured in the YAML configuration
  file. If not specified, 1 is used as default.
  (issue #252)
* Arguments to CMake/autotools can now be append to the default
  arguments with the 'configure_args+' key in the YAML
  configuration file. Patch-files can be appended with the
  'patch_file+' key and doc-files with 'docfiles+'.
  (issue #257)

## Version 1.1.18
### modulecmd
* BUGFIX: output of load hints fixed
  (issue #241)

### build-system
* BUGFIX: parsing arguments fixed (issue #238)
* BUGFIX: group dependencies specified with the option '--with' were ignored
  (issue #236)
* BUGFIX: writing list of 'systems' to module configuration file fixed
  (issues #235, #243)
* Files in `$PMODULES_TMPDIR/<module>` are now removed before
  and after building a module (except cleanup has been disbaled).
  (issues #242, #245)
* Set prefix and directory/name of modulefile based on group not on
  environment variables like `{COMPILER,MPI,HDF5}_VERSION`
  (issue #244)

### Toolchain
* update to Tcl 8.6.14 (issue #239)

### General
* link to documentation added to usage/help text (issue #237)

## Version 1.1.17
### build-system
**Fixes**
* usage output reviewed
* compiling in source tree fixed if YAML config file is used

## Version 1.1.16

### modulecmd
**User visible changes**
* YAML module configuration file. Configured in this file are
  * the release stage
  * the list of systems the module is available
  * the list of systems the module shadowed

  Wherby the items in list of systems are either
  * hostnames or glob style patterns of hostnames (e.g. merlin-*)
  * OS names like rhel7
  
  The filename is `.config-<version>` and is located in the same directory as the modulefile.
  For now the `.release-<version>` files are still used if no YAML configuration file exist.

### build-system
**User visible changes**
* YAML build configuration files. For now `variants` files are still supported.

---

## Version 1.1.15

### modulecmd
**User visible changes**
* prevent loading of a module on dedicated systems via blocklist

### build-system
**User visible changes**
* the function to compare versions can now be called with a single
  argument. In this case the given version is compared to version
  of currently build module.

**Internal changes and fixes***
* bugfix: due to a bug in compiling the list of (legacy) config files,
  each module was build twice.

---

## Version 1.1.14

### build-system 
**Internal changes and fixes**
* lookup of default legacy config files fixed

---

## Version 1.1.13

### modulecmd
**User visible changes**
* collections
* shadowing modules with an overlay
* Lmod support
* argument/option handling fixed
* using groups/extending MODULEPATH fixed (module use ...)

**Internal changes and fixes**
* use Lua and Lmod from Pmodules
* std::upvar() replaced
* Bugfixes
---

## Version 1.1.12

### modulecmd
**User visible changes**
* experimental support for Lmod
* support for use flags has been removed.
  (A postfix like '_slurm' can still be used)

---

## Version 1.1.11

### modulecmd
**User visible changes**
* Improved error handling and messages
* Python support added

**Internal changes and fixes**
* Bugfixes

### build-system
**User visible changes**
* building with legacy variants files fixed.
* building with YAML config files is still experimental
  and not recommended for now

---

## Version 1.1.10 

### modulecmd
**User visible changes**
* New options for `module search`.
  * With the option `--group` the search can be restricted to a
    group.
  * With the option `--newest` only the newest versions are
    displayed.
* `find` as alias for the sub-command `search` added.
* Bugfix: the sub-commands `whatis` and `keyword|apropos` were
  broken by design. 
* Bugfix: after loading a `Pmodules` module, it was not shown with
  `module list`.
* Bugfix in scanning the depth of groups.
* Bugfix: after `module purge` the environment variable
  `PMODULES_HOME` was not defined an more.
* Bugfix: source the shell init file only if a `Pmodules` module
  is loaded.
* Bugfix: unsetting aliases in modulefiles was not handled
  properly in `module purge`

**Internal changes and fixes**
* initialisation error for bash and zsh fixed

### build-system
**User visible changes**
* `modbuild` is now defined as function like `module`. Therefor no
  `Pmodules` module must be loaded to build a module with `modbuild`
* The system can now be defined in the module (YAML) configuration
  file.
* Build dependencies can (and should) now be specified with
  `build_requires` in the YAML configuration file.
* Bugfix: cleanup of modulefiles in overlays fixed. A module can be in
  more than one overlay. These overlays must be specified in the
  module configuration file. 
* Bugfix: querying dependencies from YAML configuration file
  fixed. Under some conditions the string 'null' was in the list
  of dependencies.
* Bugfix: create group directory if it doesn't exist.
* Bugfix: create the module `$PREFIX` before processing the
  install targets not before all targets. If `$PREFIX` is created
  before processing any target and the build fails, `modbuild`
  assumes that the module have been already built successfully.

**Internal changes and fixes**
* code review/re-factoring
* `modbuild` is now using the Bash installed in `Pmodules` itself.
* test code with `set -o nounset`, several issues with this
  setting fixed (not necessarily bugs).

**other changes**
* The build script to bootstrap Pmodules itself doesn't use modbuild
  any more to compile required software packages.  With this change
  we can remove some special cases from modbuild.
* The bootstrap script requires Bash 5.0 or newer now.
* Bugfix: in the `Pmodules` modulefile force the sourcing of the
  shell init script while in mode `load` only.

---

## Version 1.1.9

### modulecmd
**User visible changes**
* Overlay info added to output of sub-command `search`.
* Output of `module search --verbose` revised for better readability.

**Internal changes and fixes**
* The shell`s init file is sourced, when Pmodules is loaded as module.
  This is required if there are changes in the module function or too
  define new shell functions.
* A bug in `libmodules.tcl:module-addgroup()` which crashed 
  `module load ...` has been fixed.
* In versions before 1.1.9 a colon at the beginning or end of `MODULEPATH`
  crashed the module function. This has been fixed. 

### build-system
**User visible changes**
* The command `modbuild` is now defined as shell function analog to
  the `module` command. The main reason to introduce this function
  is due to the fact that Bash version 5 or newer is now required
  by `modbuild`. The function `modbuild` load Bash 5.x as module
  before calling the modbuild-script. If you want to use the script
  directly, a Bash binary with version 5.x must be in PATH.
* If a build-script is in the current working directory,
  `modbuild` can now be called without specifying the build-script.
* In case of an error in a build-step the build process did not 
  abort as it should. This has been fixed.
* The option `--overlay` can now be used
  - to define an overlay if legacy variants files are used
  - to override the overlay in a YAML variants file.
* The new keyword `with` has been introduced in YAML variants file
  to specified hierarchical dependencies.
* The function `pbuild::supported_os` has been
  removed. `pbuild::supported_systems` provides the same
  functionality for legacy configuration files. In YAML module
  configuration files `systems` have to be used.

**Internal changes and fixes**
* bugfix in setting `PATH`
* requires bash 5 or later

---

## Version 1.1.8

### modulecmd
**User visible changes**
* configuration in YAML files
* modulefiles and software must not
  have a common root directory
* the installation root must be specified, it doesn`t default
  to the base 'overlay' any more.
* zsh initialisation fixed.

**Internal changes and fixes**
* std::upvar() replaced with reference variables in part of the 
  code.
* environment variable `PMODULES_ROOT` removed.
* unsetting aliases fixed.
* update to bash 5.1.16
* update to findutils 4.9 (macOS only)
* minor fixes

### build-system
**User visible changes**
* YAML format for variants files
	
**Internal changes and fixes**
* use lib `libpmodules.bash`
* bugfixes

### modmanage
**User visible changes**
* none, support for overlays still missing

**Internal changes and fixes**
* none

---

## Version 1.1.7
### modulecmd
* list of available overlays in subcommand `use` is now better readable

### buid-system
* overlay definition must now be in YAML format
* support for YAML formatted variant files (the legacy format 
  is still supported)
* build-system in 1.1.6 was still work in progress and broken

---

## Version 1.1.6
### modulecmd
* bugfix in searching/loading modules in a hierarchical
  group

---

## Version 1.1.5
### modulecmd
* first public version with the overlay feature

---

## Version 1.0.0rc11
### modulecmd
**User visible changes**
* handling of set-alias in modulefile fixed
---

## Version 1.0.0rc10
### modulecmd
**User visible changes**
* The term "releases" has been replaced with "release stages". 
  The visible changes are the change of the option 
  `--all-releases` to `--all-release-stages`, adapted
  help text and configuration files.
* New configuration file `Pmodules.conf` to configure the 
  default visible group, the default visible release stages
  and the defined releases stages. These information has
  been stripped from the profiles `profiles.{bash,csh,zsh}`.
* module are now sorted numerically in output of
  `module avail`.
* option `-?` added as alias for `--help`.
* new option `--glob` for `module search`. This enables 
  shell glob-pattern searches.
**Internal changes and fixes**
* bugfix in removing temp-file in exit function.
* terse output of `module avail` fixed.
* broken help for sub-commands fixed.
* missing group in output of `module avail` fixed.
* broken output of `module search --print-modulefiles` fixed.
* argument handling fixed after `--`.
* cleanup option/argument handling.

### build-system
**User visible changes**
* none
**Internal changes and fixes**
* bugfixes in the functions `pbuild::version_{le,gt}
* bugfix in recognising newer CentOS versions

### modmanage
**User visible changes**
* complete re-implementation of (broken) modmanage
---

## Version 1.0.0rc9
### modulecmd
**User visible changes**
* a Pmodules module must be the first module loaded
* new option `--group|-g GROUP` to list available modules in `GROUP`
* align columns in output of `module avail`
* remove path to Pmodules bin directory while unloading a Pmodules module
* exclude a Pmodules module from being purged 
* follow sym-links in `ROOT/GROUP/modulefiles`

**Internal changes and fixes**
* use default field separator by unsetting `IFS`
* use read-only variables for all used commands with full path  
* better tmp-file creation/deletion
* more bugfixes

### build-system
**User visible changes***
* group hierarchy can now be defined in a config file

**Internal changes and fixes**
* more bugfixes

---

## Version 1.0.0rc8
### modulecmd
**User visible changes**
* Pmodules can now be loaded as module
* Since `${PMODULES_HOME}/bin` has been removed from `PATH` in `1.0.0rc7` a Pmodules
  module must be loaded to make the build system available.

**Internal changes and fixes**
* use system binaries in `/bin:/usr/bin` if possible
---

## Version 1.0.0rc7
### modulecmd
**User visible changes**
* add options to `module search` to show dependencies

**Internal changes and fixes**
* hardcoded path in `profile.csh`fixed
* bugfixes
* Update to BASH 5.1
* Update to Tcl 8.6.10

### build-system
**User visible changes**
* building deprecated modules must be forced
* support for versioned modulefiles in build-blocks: if a modulefile 
  `modulefile-X[.Y[.Z]]` exists in the build-block it will be taken
  in favour of `modulefile`-
* bootstrap/build script reviewed, `--config` option removed, help for all  
  sub-commands added

**Internal changes and fixes**
* installation of fallback shared libraries fixed.
* bugfixes

### modmanage
**User visible changes**
* Support of run-time dependencies which are required but must not be loaded
---

## Version 1.0.0rc6
### modulecmd
- Support for shell `sh` added (#86, #90).
- Broken 'module load'  with (T)CSH fixed (#88).
- prepend instead of append Pmodules bin directory to `PATH` (#87).

### build-system
- Support added for a wildcard (`.*`) version as argument to `modbuild` (#78)
- Handle empty list of be installed shared libraries fixed (#89).

### Building and installing Pmodules
- Bugs in bootstrapping Pmodules fixed (#82)
---

## Version 1.0.0rc5 (since 1.0.0rc2)
### modulecmd
- log `module load` commands to system logger (#80)
- Bugs fixed in printing load hints (#48, #49)
- `--with` option of sub-command search now accepts a comma separated list of strings
- `PMODULES_ENV` is exported only on content changes
- more bugfixes

### build-system
- The argument passed with the option `--system` is not any more a synonym for
  the kernel of the system (like Linux, Darwin). It now defines a target operating
  system like RHEL6, macOS1014 etc (#72).
- calling `pbuild::make_all` in a build-script is now deprecate
- bugfixes
---

## 1.0.0rc4
- never tagged
---

## 1.0.0rc3
- never tagged