add a tools page for merlin7

_data/sidebars/merlin7_sidebar.yml

@@ -44,6 +44,8 @@ entries:
             url: /merlin7/using-modules.html
           - title: Software repository - Spack
             url: /merlin7/spack.html
+          - title: General Tools
+            url: /merlin7/tools.html
       - title: Slurm General Documentation
         folderitems:
           - title: Merlin7 Infrastructure

pages/merlin7/02-How-To-Use-Merlin/merlin_tools.md

@@ -0,0 +1,108 @@
+---
+title: Merlin7 Tools
+#tags:
+keywords: merlin_quotas
+#last_updated: 07 September 2022
+#summary: ""
+sidebar: merlin7_sidebar
+permalink: /merlin7/tools.html
+---
+
+## About
+
+We provide tool(s) to help user get the most out of using the cluster. The tools
+described here are organised by use case and include usage examples.
+
+## Files and Directories
+
+### `merlin_quotas`
+
+This tool is available on all of the login nodes and provides a brief overview of
+a user's filesystem quotas. These are limits which restrict how much storage (or
+number of files) a user can create. A generic table of filesystem quotas can be
+found on the [Storage page](/merlin7/storage.html#dir_classes).
+
+#### Example #1: Viewing quotas
+
+Simply calling `merlin_quotas` will show you a table of our quotas:
+
+```console
+$ merlin_quotas 
+Path            SpaceUsed  SpaceQuota  Space %  FilesUsed  FilesQuota  Files %
+--------------  ---------  ----------  -------  ---------  ----------  -------
+/data/user      30.26G     1T          03%      367296     2097152     18%    
+ └─ <USERNAME>
+/afs/psi.ch     3.4G       9.5G        36%      0          0           0%     
+ └─ user/<USERDIR>                                                                
+/data/project   2.457T     10T         25%      58         2097152     00%    
+ └─ bio/shared                                                                
+/data/project   338.3G     10T         03%      199391     2097152     10%    
+ └─ bio/hpce 
+```
+
+{{site.data.alerts.tip}}You can change the width of the table by either passing
+<code>--no-wrap</code> (to disable wrapping of the <i>Path</i>) or <code>--width N</code>
+(to explicitly set some width by <code>N</code> characters).
+{{site.data.alerts.end}}
+
+#### Example #2: Project view
+
+The tool can also be used to list out information about what projects directories
+there are and who owns/manages these:
+
+```console
+$ merlin_quotas projects
+Project ID  Path                      Owner      Group
+----------  ------------------------  ---------  --------------
+600000000   /data/project/bio/shared  germann_e  unx-merlin_adm
+600000001   /data/project/bio/hpce    assman_g   unx-merlin_adm
+```
+
+By default this only shows information on projects that you have access to, but
+to view the whole list you can pass `--all` flag:
+
+```console
+$ merlin_quotas projects --all
+Project ID  Path                             Owner           Group
+----------  -------------------------------  --------------  -----------------
+500000000   /data/project/general/mcnp       gac-mcnp        unx-mcnp_all
+500000001   /data/project/general/vis_as     talanov_v       unx-vis_as
+500000002   /data/project/general/mmm        krack           org-7302
+500000003   /data/project/general            laeuch_a        org-7201
+             └─ LTC_CompPhys
+600000000   /data/project/bio/shared         germann_e       unx-merlin_adm
+600000001   /data/project/bio/hpce           assman_g        unx-merlin_adm
+600000002   /data/project/bio/abrahams       abrahams_j      unx-bio_abrahams
+600000003   /data/project/bio/benoit         benoit_r        unx-bio_benoit
+600000004   /data/project/bio/ishikawa       ishikawa        unx-bio_ishikawa
+600000005   /data/project/bio/kammerer       kammerer_r      unx-bio_kammerer
+600000006   /data/project/bio/korkhov        korkhov_v       unx-bio_korkhov
+600000007   /data/project/bio/luo            luo_j           unx-bio_luo
+600000008   /data/project/bio/mueller        mueller_e       unx-bio_mueller
+600000009   /data/project/bio/poghosyan      poghosyan_e     unx-bio_poghosyan
+600000010   /data/project/bio/schertler      schertler_g     unx-bio_schertler
+600000011   /data/project/bio/shivashankar   shivashankar_g  unx-bio_shivashan
+600000012   /data/project/bio/standfuss      standfuss       unx-bio_standfuss
+600000013   /data/project/bio/steinmetz      steinmetz       unx-bio_steinmetz
+```
+
+{{site.data.alerts.tip}}As above you can change the table width by pass either
+<code>--no-wrap</code> or <code>--width N</code>.
+{{site.data.alerts.end}}
+
+#### Example #3: Project config
+
+To make tracking quotas of projects easier, `merlin_quotas` generates a config
+file in your home directory which defines the projects to show when you call the
+tool (called `~/.merlin_quotas`).
+
+The config file simply contains a list (one per line) of project IDs which should
+be tracked. In theory any (or all available projects) can be tracked, but due to
+UNIX and Lustre permissions, accessing quotas information for a project you're not
+a member of **is not possible**.
+
+If you are added/removed from a project, you can update this config file by
+calling `merlin_quotas genconf --force` (notice the `--force`, which will overwrite
+your existing config file) or by editing the file by hand (*not recommended*).
+
+

pages/merlin7/02-How-To-Use-Merlin/storage.md

@@ -28,31 +28,30 @@ Some of the Merlin7 directories have quotas applied. A way for checking the quot
 This command is useful to show all quotas for the different user storage directories and partitions (including AFS). To check your quotas, please run:
 
 ```console
-$ merlin_quotas
+$ merlin_quotas 
 Path            SpaceUsed  SpaceQuota  Space %  FilesUsed  FilesQuota  Files %
 --------------  ---------  ----------  -------  ---------  ----------  -------
-/data/user      29.85G     1T          0%       366671     2097152     0%
+/data/user      30.26G     1T          03%      367296     2097152     18%    
  └─ <USERNAME>
-/afs/psi.ch     3.4G       9.5G        36%      0          0           0%
+/afs/psi.ch     3.4G       9.5G        36%      0          0           0%     
- └─ user/v/<USERNAME>
+ └─ user/<USERDIR>                                                                
-/data/scratch   680.1M     512G        0%       366897     0           0%
+/data/project   2.457T     10T         25%      58         2097152     00%    
- └─ shared
+ └─ bio/shared                                                                
-/data/project   1.115T     10T         0%       50         2097152     0%
+/data/project   338.3G     10T         03%      199391     2097152     10%    
- └─ bio/shared
+ └─ bio/hpce 
 ```
 
+{{site.data.alerts.note}}On first use you will see a message about some configuration being generated, this is expected. Don't be
+surprised that it takes some time. After this using <code>merlin_quotas</code> should be faster.
+{{site.data.alerts.end}}
+
 The output shows the quotas set and how much you are using of the quota, for each filesystem that has this set. Notice that some users will have
 one or more `/data/project/...` directories showing, depending on whether you are part of a specific PSI research group or project.
 
-{{site.data.alerts.tip}}If you're using <code>merlin_quotas</code> for the first time, your project quotas might not appear. You can fix this by
+The general quota constraints for the different directories are shown in the [table below](#dir_classes). Further details on how to use `merlin_quotas`
-running <code>merlin_quotes --gen-project-config --use-groups</code> which will generate in your home directory a <code>~/.merlin_quotas</code> file
+can be found on the [Tools page](/merlin7/tools.html).
-with all the projects you have access to. This file can also be hand modified, keep in mind though that security restrictions are in place, so you
-won't be able to looking into project data directories that you are not a member of.
-{{site.data.alerts.end}}
 
-The general quota constraints for the different directories are shown in the table below.
+{{site.data.alerts.tip}}If you're interesting, you can retrieve the Lustre-based quota information directly by calling
-
-{{site.data.alerts.note}}If you're interesting, you can retrieve the Lustre-based quota information directly by calling
 <code>lfs quota -h -p $(( 100000000 + $(id -u $USER) )) /data</code> directly. Using the <code>merlin_quotas</code> command is more
 convenient and shows all your relevant filesystem quotas.
 {{site.data.alerts.end}}
@@ -75,7 +74,7 @@ Once the <b>grace time</b> or hard limit are reached, users will be unable to wr
 when this is possible, see below table).
 {{site.data.alerts.end}}
 
-Properties of the directory classes:
+<a name="dir_classes"></a>Properties of the directory classes:
 
 | Directory                          | Block Quota [Soft:Hard] | Inode Quota [Soft:Hard] | GraceTime |  Quota Change Policy: Block        | Quota Change Policy: Inodes      | Backup |
 | ---------------------------------- | ----------------------- | ----------------------- | :-------: | :--------------------------------- |:-------------------------------- | ------ |
@@ -113,7 +112,7 @@ Home directory quotas are defined in a per Lustre project basis. The quota can b
 ### Project data directory
 
 This storage is intended for keeping large amounts of a project's data, where the data also can be
-shared by all members of the project (the project's corresponding unix group). We recommend to keep most data in
+shared by all members of the project (the project's corresponding UNIX group). We recommend to keep most data in
 project related storage spaces, since it allows users to coordinate. Also, project spaces have more flexible policies
 regarding extending the available storage space.