add a tools page for merlin7
This commit is contained in:
parent
8ba2b715cd
commit
8cf6db43f3
GPG Key ID:
0C24C120CDED56F0
@ -44,6 +44,8 @@ entries:
|
||||
url: /merlin7/using-modules.html
|
||||
- title: Software repository - Spack
|
||||
url: /merlin7/spack.html
|
||||
- title: General Tools
|
||||
url: /merlin7/tools.html
|
||||
- title: Slurm General Documentation
|
||||
folderitems:
|
||||
- title: Merlin7 Infrastructure
|
||||
|
||||
108
pages/merlin7/02-How-To-Use-Merlin/merlin_tools.md
Normal file
108
pages/merlin7/02-How-To-Use-Merlin/merlin_tools.md
Normal file
@ -0,0 +1,108 @@
|
||||
---
|
||||
title: Merlin7 Tools
|
||||
#tags:
|
||||
keywords: merlin_quotas
|
||||
#last_updated: 07 September 2022
|
||||
#summary: ""
|
||||
sidebar: merlin7_sidebar
|
||||
permalink: /merlin7/tools.html
|
||||
---
|
||||
|
||||
## About
|
||||
|
||||
We provide tool(s) to help user get the most out of using the cluster. The tools
|
||||
described here are organised by use case and include usage examples.
|
||||
|
||||
## Files and Directories
|
||||
|
||||
### `merlin_quotas`
|
||||
|
||||
This tool is available on all of the login nodes and provides a brief overview of
|
||||
a user's filesystem quotas. These are limits which restrict how much storage (or
|
||||
number of files) a user can create. A generic table of filesystem quotas can be
|
||||
found on the [Storage page](/merlin7/storage.html#dir_classes).
|
||||
|
||||
#### Example #1: Viewing quotas
|
||||
|
||||
Simply calling `merlin_quotas` will show you a table of our quotas:
|
||||
|
||||
```console
|
||||
$ merlin_quotas
|
||||
Path SpaceUsed SpaceQuota Space % FilesUsed FilesQuota Files %
|
||||
-------------- --------- ---------- ------- --------- ---------- -------
|
||||
/data/user 30.26G 1T 03% 367296 2097152 18%
|
||||
└─ <USERNAME>
|
||||
/afs/psi.ch 3.4G 9.5G 36% 0 0 0%
|
||||
└─ user/<USERDIR>
|
||||
/data/project 2.457T 10T 25% 58 2097152 00%
|
||||
└─ bio/shared
|
||||
/data/project 338.3G 10T 03% 199391 2097152 10%
|
||||
└─ bio/hpce
|
||||
```
|
||||
|
||||
{{site.data.alerts.tip}}You can change the width of the table by either passing
|
||||
<code>--no-wrap</code> (to disable wrapping of the <i>Path</i>) or <code>--width N</code>
|
||||
(to explicitly set some width by <code>N</code> characters).
|
||||
{{site.data.alerts.end}}
|
||||
|
||||
#### Example #2: Project view
|
||||
|
||||
The tool can also be used to list out information about what projects directories
|
||||
there are and who owns/manages these:
|
||||
|
||||
```console
|
||||
$ merlin_quotas projects
|
||||
Project ID Path Owner Group
|
||||
---------- ------------------------ --------- --------------
|
||||
600000000 /data/project/bio/shared germann_e unx-merlin_adm
|
||||
600000001 /data/project/bio/hpce assman_g unx-merlin_adm
|
||||
```
|
||||
|
||||
By default this only shows information on projects that you have access to, but
|
||||
to view the whole list you can pass `--all` flag:
|
||||
|
||||
```console
|
||||
$ merlin_quotas projects --all
|
||||
Project ID Path Owner Group
|
||||
---------- ------------------------------- -------------- -----------------
|
||||
500000000 /data/project/general/mcnp gac-mcnp unx-mcnp_all
|
||||
500000001 /data/project/general/vis_as talanov_v unx-vis_as
|
||||
500000002 /data/project/general/mmm krack org-7302
|
||||
500000003 /data/project/general laeuch_a org-7201
|
||||
└─ LTC_CompPhys
|
||||
600000000 /data/project/bio/shared germann_e unx-merlin_adm
|
||||
600000001 /data/project/bio/hpce assman_g unx-merlin_adm
|
||||
600000002 /data/project/bio/abrahams abrahams_j unx-bio_abrahams
|
||||
600000003 /data/project/bio/benoit benoit_r unx-bio_benoit
|
||||
600000004 /data/project/bio/ishikawa ishikawa unx-bio_ishikawa
|
||||
600000005 /data/project/bio/kammerer kammerer_r unx-bio_kammerer
|
||||
600000006 /data/project/bio/korkhov korkhov_v unx-bio_korkhov
|
||||
600000007 /data/project/bio/luo luo_j unx-bio_luo
|
||||
600000008 /data/project/bio/mueller mueller_e unx-bio_mueller
|
||||
600000009 /data/project/bio/poghosyan poghosyan_e unx-bio_poghosyan
|
||||
600000010 /data/project/bio/schertler schertler_g unx-bio_schertler
|
||||
600000011 /data/project/bio/shivashankar shivashankar_g unx-bio_shivashan
|
||||
600000012 /data/project/bio/standfuss standfuss unx-bio_standfuss
|
||||
600000013 /data/project/bio/steinmetz steinmetz unx-bio_steinmetz
|
||||
```
|
||||
|
||||
{{site.data.alerts.tip}}As above you can change the table width by pass either
|
||||
<code>--no-wrap</code> or <code>--width N</code>.
|
||||
{{site.data.alerts.end}}
|
||||
|
||||
#### Example #3: Project config
|
||||
|
||||
To make tracking quotas of projects easier, `merlin_quotas` generates a config
|
||||
file in your home directory which defines the projects to show when you call the
|
||||
tool (called `~/.merlin_quotas`).
|
||||
|
||||
The config file simply contains a list (one per line) of project IDs which should
|
||||
be tracked. In theory any (or all available projects) can be tracked, but due to
|
||||
UNIX and Lustre permissions, accessing quotas information for a project you're not
|
||||
a member of **is not possible**.
|
||||
|
||||
If you are added/removed from a project, you can update this config file by
|
||||
calling `merlin_quotas genconf --force` (notice the `--force`, which will overwrite
|
||||
your existing config file) or by editing the file by hand (*not recommended*).
|
||||
|
||||
|
||||
@ -28,31 +28,30 @@ Some of the Merlin7 directories have quotas applied. A way for checking the quot
|
||||
This command is useful to show all quotas for the different user storage directories and partitions (including AFS). To check your quotas, please run:
|
||||
|
||||
```console
|
||||
$ merlin_quotas
|
||||
$ merlin_quotas
|
||||
Path SpaceUsed SpaceQuota Space % FilesUsed FilesQuota Files %
|
||||
-------------- --------- ---------- ------- --------- ---------- -------
|
||||
/data/user 29.85G 1T 0% 366671 2097152 0%
|
||||
/data/user 30.26G 1T 03% 367296 2097152 18%
|
||||
└─ <USERNAME>
|
||||
/afs/psi.ch 3.4G 9.5G 36% 0 0 0%
|
||||
└─ user/v/<USERNAME>
|
||||
/data/scratch 680.1M 512G 0% 366897 0 0%
|
||||
└─ shared
|
||||
/data/project 1.115T 10T 0% 50 2097152 0%
|
||||
└─ bio/shared
|
||||
/afs/psi.ch 3.4G 9.5G 36% 0 0 0%
|
||||
└─ user/<USERDIR>
|
||||
/data/project 2.457T 10T 25% 58 2097152 00%
|
||||
└─ bio/shared
|
||||
/data/project 338.3G 10T 03% 199391 2097152 10%
|
||||
└─ bio/hpce
|
||||
```
|
||||
|
||||
{{site.data.alerts.note}}On first use you will see a message about some configuration being generated, this is expected. Don't be
|
||||
surprised that it takes some time. After this using <code>merlin_quotas</code> should be faster.
|
||||
{{site.data.alerts.end}}
|
||||
|
||||
The output shows the quotas set and how much you are using of the quota, for each filesystem that has this set. Notice that some users will have
|
||||
one or more `/data/project/...` directories showing, depending on whether you are part of a specific PSI research group or project.
|
||||
|
||||
{{site.data.alerts.tip}}If you're using <code>merlin_quotas</code> for the first time, your project quotas might not appear. You can fix this by
|
||||
running <code>merlin_quotes --gen-project-config --use-groups</code> which will generate in your home directory a <code>~/.merlin_quotas</code> file
|
||||
with all the projects you have access to. This file can also be hand modified, keep in mind though that security restrictions are in place, so you
|
||||
won't be able to looking into project data directories that you are not a member of.
|
||||
{{site.data.alerts.end}}
|
||||
The general quota constraints for the different directories are shown in the [table below](#dir_classes). Further details on how to use `merlin_quotas`
|
||||
can be found on the [Tools page](/merlin7/tools.html).
|
||||
|
||||
The general quota constraints for the different directories are shown in the table below.
|
||||
|
||||
{{site.data.alerts.note}}If you're interesting, you can retrieve the Lustre-based quota information directly by calling
|
||||
{{site.data.alerts.tip}}If you're interesting, you can retrieve the Lustre-based quota information directly by calling
|
||||
<code>lfs quota -h -p $(( 100000000 + $(id -u $USER) )) /data</code> directly. Using the <code>merlin_quotas</code> command is more
|
||||
convenient and shows all your relevant filesystem quotas.
|
||||
{{site.data.alerts.end}}
|
||||
@ -75,7 +74,7 @@ Once the <b>grace time</b> or hard limit are reached, users will be unable to wr
|
||||
when this is possible, see below table).
|
||||
{{site.data.alerts.end}}
|
||||
|
||||
Properties of the directory classes:
|
||||
<a name="dir_classes"></a>Properties of the directory classes:
|
||||
|
||||
| Directory | Block Quota [Soft:Hard] | Inode Quota [Soft:Hard] | GraceTime | Quota Change Policy: Block | Quota Change Policy: Inodes | Backup |
|
||||
| ---------------------------------- | ----------------------- | ----------------------- | :-------: | :--------------------------------- |:-------------------------------- | ------ |
|
||||
@ -113,7 +112,7 @@ Home directory quotas are defined in a per Lustre project basis. The quota can b
|
||||
### Project data directory
|
||||
|
||||
This storage is intended for keeping large amounts of a project's data, where the data also can be
|
||||
shared by all members of the project (the project's corresponding unix group). We recommend to keep most data in
|
||||
shared by all members of the project (the project's corresponding UNIX group). We recommend to keep most data in
|
||||
project related storage spaces, since it allows users to coordinate. Also, project spaces have more flexible policies
|
||||
regarding extending the available storage space.
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user