diff --git a/_data/sidebars/merlin7_sidebar.yml b/_data/sidebars/merlin7_sidebar.yml index df6f9f7..8636c90 100644 --- a/_data/sidebars/merlin7_sidebar.yml +++ b/_data/sidebars/merlin7_sidebar.yml @@ -44,6 +44,8 @@ entries: url: /merlin7/using-modules.html - title: Software repository - Spack url: /merlin7/spack.html + - title: General Tools + url: /merlin7/tools.html - title: Slurm General Documentation folderitems: - title: Merlin7 Infrastructure diff --git a/pages/merlin7/02-How-To-Use-Merlin/merlin_tools.md b/pages/merlin7/02-How-To-Use-Merlin/merlin_tools.md new file mode 100644 index 0000000..93a6af3 --- /dev/null +++ b/pages/merlin7/02-How-To-Use-Merlin/merlin_tools.md @@ -0,0 +1,108 @@ +--- +title: Merlin7 Tools +#tags: +keywords: merlin_quotas +#last_updated: 07 September 2022 +#summary: "" +sidebar: merlin7_sidebar +permalink: /merlin7/tools.html +--- + +## About + +We provide tool(s) to help user get the most out of using the cluster. The tools +described here are organised by use case and include usage examples. + +## Files and Directories + +### `merlin_quotas` + +This tool is available on all of the login nodes and provides a brief overview of +a user's filesystem quotas. These are limits which restrict how much storage (or +number of files) a user can create. A generic table of filesystem quotas can be +found on the [Storage page](/merlin7/storage.html#dir_classes). + +#### Example #1: Viewing quotas + +Simply calling `merlin_quotas` will show you a table of our quotas: + +```console +$ merlin_quotas +Path SpaceUsed SpaceQuota Space % FilesUsed FilesQuota Files % +-------------- --------- ---------- ------- --------- ---------- ------- +/data/user 30.26G 1T 03% 367296 2097152 18% + └─ +/afs/psi.ch 3.4G 9.5G 36% 0 0 0% + └─ user/ +/data/project 2.457T 10T 25% 58 2097152 00% + └─ bio/shared +/data/project 338.3G 10T 03% 199391 2097152 10% + └─ bio/hpce +``` + +{{site.data.alerts.tip}}You can change the width of the table by either passing +--no-wrap (to disable wrapping of the Path) or --width N +(to explicitly set some width by N characters). +{{site.data.alerts.end}} + +#### Example #2: Project view + +The tool can also be used to list out information about what projects directories +there are and who owns/manages these: + +```console +$ merlin_quotas projects +Project ID Path Owner Group +---------- ------------------------ --------- -------------- +600000000 /data/project/bio/shared germann_e unx-merlin_adm +600000001 /data/project/bio/hpce assman_g unx-merlin_adm +``` + +By default this only shows information on projects that you have access to, but +to view the whole list you can pass `--all` flag: + +```console +$ merlin_quotas projects --all +Project ID Path Owner Group +---------- ------------------------------- -------------- ----------------- +500000000 /data/project/general/mcnp gac-mcnp unx-mcnp_all +500000001 /data/project/general/vis_as talanov_v unx-vis_as +500000002 /data/project/general/mmm krack org-7302 +500000003 /data/project/general laeuch_a org-7201 + └─ LTC_CompPhys +600000000 /data/project/bio/shared germann_e unx-merlin_adm +600000001 /data/project/bio/hpce assman_g unx-merlin_adm +600000002 /data/project/bio/abrahams abrahams_j unx-bio_abrahams +600000003 /data/project/bio/benoit benoit_r unx-bio_benoit +600000004 /data/project/bio/ishikawa ishikawa unx-bio_ishikawa +600000005 /data/project/bio/kammerer kammerer_r unx-bio_kammerer +600000006 /data/project/bio/korkhov korkhov_v unx-bio_korkhov +600000007 /data/project/bio/luo luo_j unx-bio_luo +600000008 /data/project/bio/mueller mueller_e unx-bio_mueller +600000009 /data/project/bio/poghosyan poghosyan_e unx-bio_poghosyan +600000010 /data/project/bio/schertler schertler_g unx-bio_schertler +600000011 /data/project/bio/shivashankar shivashankar_g unx-bio_shivashan +600000012 /data/project/bio/standfuss standfuss unx-bio_standfuss +600000013 /data/project/bio/steinmetz steinmetz unx-bio_steinmetz +``` + +{{site.data.alerts.tip}}As above you can change the table width by pass either +--no-wrap or --width N. +{{site.data.alerts.end}} + +#### Example #3: Project config + +To make tracking quotas of projects easier, `merlin_quotas` generates a config +file in your home directory which defines the projects to show when you call the +tool (called `~/.merlin_quotas`). + +The config file simply contains a list (one per line) of project IDs which should +be tracked. In theory any (or all available projects) can be tracked, but due to +UNIX and Lustre permissions, accessing quotas information for a project you're not +a member of **is not possible**. + +If you are added/removed from a project, you can update this config file by +calling `merlin_quotas genconf --force` (notice the `--force`, which will overwrite +your existing config file) or by editing the file by hand (*not recommended*). + + diff --git a/pages/merlin7/02-How-To-Use-Merlin/storage.md b/pages/merlin7/02-How-To-Use-Merlin/storage.md index e5c29f2..a03f518 100644 --- a/pages/merlin7/02-How-To-Use-Merlin/storage.md +++ b/pages/merlin7/02-How-To-Use-Merlin/storage.md @@ -28,31 +28,30 @@ Some of the Merlin7 directories have quotas applied. A way for checking the quot This command is useful to show all quotas for the different user storage directories and partitions (including AFS). To check your quotas, please run: ```console -$ merlin_quotas +$ merlin_quotas Path SpaceUsed SpaceQuota Space % FilesUsed FilesQuota Files % -------------- --------- ---------- ------- --------- ---------- ------- -/data/user 29.85G 1T 0% 366671 2097152 0% +/data/user 30.26G 1T 03% 367296 2097152 18% └─ -/afs/psi.ch 3.4G 9.5G 36% 0 0 0% - └─ user/v/ -/data/scratch 680.1M 512G 0% 366897 0 0% - └─ shared -/data/project 1.115T 10T 0% 50 2097152 0% - └─ bio/shared +/afs/psi.ch 3.4G 9.5G 36% 0 0 0% + └─ user/ +/data/project 2.457T 10T 25% 58 2097152 00% + └─ bio/shared +/data/project 338.3G 10T 03% 199391 2097152 10% + └─ bio/hpce ``` +{{site.data.alerts.note}}On first use you will see a message about some configuration being generated, this is expected. Don't be +surprised that it takes some time. After this using merlin_quotas should be faster. +{{site.data.alerts.end}} + The output shows the quotas set and how much you are using of the quota, for each filesystem that has this set. Notice that some users will have one or more `/data/project/...` directories showing, depending on whether you are part of a specific PSI research group or project. -{{site.data.alerts.tip}}If you're using merlin_quotas for the first time, your project quotas might not appear. You can fix this by -running merlin_quotes --gen-project-config --use-groups which will generate in your home directory a ~/.merlin_quotas file -with all the projects you have access to. This file can also be hand modified, keep in mind though that security restrictions are in place, so you -won't be able to looking into project data directories that you are not a member of. -{{site.data.alerts.end}} +The general quota constraints for the different directories are shown in the [table below](#dir_classes). Further details on how to use `merlin_quotas` +can be found on the [Tools page](/merlin7/tools.html). -The general quota constraints for the different directories are shown in the table below. - -{{site.data.alerts.note}}If you're interesting, you can retrieve the Lustre-based quota information directly by calling +{{site.data.alerts.tip}}If you're interesting, you can retrieve the Lustre-based quota information directly by calling lfs quota -h -p $(( 100000000 + $(id -u $USER) )) /data directly. Using the merlin_quotas command is more convenient and shows all your relevant filesystem quotas. {{site.data.alerts.end}} @@ -75,7 +74,7 @@ Once the grace time or hard limit are reached, users will be unable to wr when this is possible, see below table). {{site.data.alerts.end}} -Properties of the directory classes: +Properties of the directory classes: | Directory | Block Quota [Soft:Hard] | Inode Quota [Soft:Hard] | GraceTime | Quota Change Policy: Block | Quota Change Policy: Inodes | Backup | | ---------------------------------- | ----------------------- | ----------------------- | :-------: | :--------------------------------- |:-------------------------------- | ------ | @@ -113,7 +112,7 @@ Home directory quotas are defined in a per Lustre project basis. The quota can b ### Project data directory This storage is intended for keeping large amounts of a project's data, where the data also can be -shared by all members of the project (the project's corresponding unix group). We recommend to keep most data in +shared by all members of the project (the project's corresponding UNIX group). We recommend to keep most data in project related storage spaces, since it allows users to coordinate. Also, project spaces have more flexible policies regarding extending the available storage space.