HPCE/gitea-pages
0
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Activity

vibe #3

Browse Source
This commit is contained in:
viessm_h
2025-12-11 13:30:12 +01:00
parent 84f9846a0c
commit 01ac18b3f4
24 changed files with 179 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
docs/merlin7/02-How-To-Use-Merlin/storage.md
View File

@@ -18,9 +18,11 @@ This document describes the different directories of the Merlin7 cluster.
* ***Users are responsible for backing up their own data***. Is recommended to backup the data on third party independent systems (i.e. LTS, Archive, AFS, SwitchDrive, Windows Shares, etc.).
* ***When a user leaves PSI, she or her supervisor/team are responsible to backup and move the data out from the cluster***: every few months, the storage space will be recycled for those old users who do not have an existing and valid PSI account.
{{site.data.alerts.warning}}When a user leaves PSI and their account is removed, their storage space in Merlin may be recycled.
Hence, <b>when a user leaves PSI</b>, they, their supervisor or team <b>must ensure that the data is backed up to an external storage</b>
{{site.data.alerts.end}}
!!! warning
When a user leaves PSI and their account is removed, their storage space in
Merlin may be recycled. Hence, **when a user leaves PSI**, they, their
supervisor or team **must ensure that the data is backed up to an external
storage**!
### How to check quotas
@@ -43,20 +45,22 @@ Path SpaceUsed SpaceQuota Space % FilesUsed FilesQuota Files %
└─ bio/hpce
```
{{site.data.alerts.note}}On first use you will see a message about some configuration being generated, this is expected. Don't be
surprised that it takes some time. After this using <code>merlin_quotas</code> should be faster.
{{site.data.alerts.end}}
!!! note
On first use you will see a message about some configuration being
generated, this is expected. Don't be surprised that it takes some time.
After this using `merlin_quotas` should be faster.
The output shows the quotas set and how much you are using of the quota, for each filesystem that has this set. Notice that some users will have
one or more `/data/project/...` directories showing, depending on whether you are part of a specific PSI research group or project.
The general quota constraints for the different directories are shown in the [table below](#dir_classes). Further details on how to use `merlin_quotas`
can be found on the [Tools page](/merlin7/tools.html).
can be found on the [Tools page](merlin_tools.md).
{{site.data.alerts.tip}}If you're interesting, you can retrieve the Lustre-based quota information directly by calling
<code>lfs quota -h -p $(( 100000000 + $(id -u $USER) )) /data</code> directly. Using the <code>merlin_quotas</code> command is more
convenient and shows all your relevant filesystem quotas.
{{site.data.alerts.end}}
!!! tip
If you're interesting, you can retrieve the Lustre-based quota information
directly by calling `lfs quota -h -p $(( 100000000 + $(id -u $USER) ))
/data` directly. Using the `merlin_quotas` command is more convenient and
shows all your relevant filesystem quotas.
## Merlin7 directories
@@ -70,11 +74,14 @@ Merlin7 offers the following directory classes for users:
* `/scratch`: Local *scratch* disk (only visible by the node running a job).
* `/data/scratch/shared`: Shared *scratch* disk (visible from all nodes).
{{site.data.alerts.tip}}In Lustre there is a concept called <b>grace time</b>. Filesystems have a block (amount of data) and inode (number of files) quota.
These quotas contain a soft and hard limits. Once the soft limit is reached, users can keep writing up to their hard limit quota during the <b>grace period</b>.
Once the <b>grace time</b> or hard limit are reached, users will be unable to write and will need remove data below the soft limit (or ask for a quota increase
when this is possible, see below table).
{{site.data.alerts.end}}
!!! tip
In Lustre there is a concept called **grace time**. Filesystems have a
block (amount of data) and inode (number of files) quota. These quotas
contain a soft and hard limits. Once the soft limit is reached, users can
keep writing up to their hard limit quota during the **grace period**.
Once the **grace time** or hard limit are reached, users will be unable to
write and will need remove data below the soft limit (or ask for a quota
increase when this is possible, see below table).
<a name="dir_classes"></a>Properties of the directory classes:
@@ -86,10 +93,11 @@ when this is possible, see below table).
| /data/scratch/shared | USR [512GB:2TB] | | 7d | Up to x2 when strongly justified. | Changeable when justified. | no |
| /scratch | *Undef* | *Undef* | N/A | N/A | N/A | no |
{{site.data.alerts.warning}}The use of <b>/scratch</b> and <b>/data/scratch/shared</b> areas as an extension of the quota <i>is forbidden</i>. The <b>/scratch</b> and
<b>/data/scratch/shared</b> areas <i>must not contain</i> final data. Keep in mind that <br><b><i>auto cleanup policies</i></b> in the <b>/scratch</b> and
<b>/data/scratch/shared</b> areas are applied.
{{site.data.alerts.end}}
!!! warning
The use of `/scratch` and `/data/scratch/shared` areas as an extension of
the quota *is forbidden*. The `/scratch` and `/data/scratch/shared` areas
***must not contain*** final data. Keep in mind that ***auto cleanup
policies*** in the `/scratch` and `/data/scratch/shared` areas are applied.
### User home directory
@@ -134,9 +142,10 @@ Project quotas are defined in a per Lustre project basis. Users can check the pr
lfs quota -h -p $projectid /data
```
{{site.data.alerts.warning}}Checking <b>quotas</b> for the Merlin projects is not yet possible.
In the future, a list of `projectid` will be provided, so users can check their quotas.
{{site.data.alerts.end}}
!!! warning
Checking **quotas** for the Merlin projects is not yet possible. In the
future, a list of `projectid` will be provided, so users can check their
quotas.
Directory policies:
@@ -178,7 +187,7 @@ and all tasks need to do I/O on the same temporary files.
Scratch directories policies:
* Read **[Important: Code of Conduct](/merlin7/code-of-conduct.html)** for more information about Merlin7 policies.
* Read **[Important: Code of Conduct](../01-Quick-Start-Guide/code-of-conduct.md)** for more information about Merlin7 policies.
* By default, *always* use **local** first and only use **shared** if your specific use case requires it.
* Temporary files *must be deleted at the end of the job by the user*.
* Remaining files will be deleted by the system if detected.