HPCE/gitea-pages
0
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Activity

vibe #3

Browse Source
This commit is contained in:
viessm_h
2025-12-11 13:30:12 +01:00
parent 84f9846a0c
commit 01ac18b3f4
24 changed files with 179 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
docs/merlin7/02-How-To-Use-Merlin/archive.md
View File

@@ -31,16 +31,15 @@ be indirectly copied to these (**decentral mode**).
Archiving can be done from any node accessible by the users (usually from the login nodes).
{{site.data.alerts.tip}} Archiving can be done in two different ways:
<br>
<b>'Central mode':</b> Possible for the user and project data directories, is the
fastest way as it does not require remote copy (data is directly retreived by central AIT servers from Merlin
through 'merlin-archive.psi.ch').
<br>
<br>
<b>'Decentral mode':</b> Possible for any directory, is the slowest way of archiving as it requires
to copy ('rsync') the data from Merlin to the central AIT servers.
{{site.data.alerts.end}}
!!! tip
Archiving can be done in two different ways:
* **Central mode**: Possible for the user and project data directories, is
the fastest way as it does not require remote copy (data is directly retreived
by central AIT servers from Merlin through <merlin-archive.psi.ch>).
* **Decentral mode**: Possible for any directory, is the slowest way of
archiving as it requires to copy ('rsync') the data from Merlin to the
central AIT servers.
## Procedure
@@ -76,7 +75,7 @@ have been assigned a **``p-group``** (e.g. ``p12345``) for the experiment. Other
Groups are usually assigned to a PI, and then individual user accounts are added to the group. This must be done
under user request through PSI Service Now. For existing **a-groups** and **p-groups**, you can follow the standard
central procedures. Alternatively, if you do not know how to do that, follow the Merlin7
**[Requesting extra Unix groups](/merlin7/request-account.html#requesting-extra-unix-groups)** procedure, or open
**[Requesting extra Unix groups](../01-Quick-Start-Guide/requesting-accounts.md#requesting-extra-unix-groups)** procedure, or open
a **[PSI Service Now](https://psi.service-now.com/psisp)** ticket.
### Documentation

2
docs/merlin7/02-How-To-Use-Merlin/connect-from-windows.md
View File

@@ -26,7 +26,7 @@ If they are missing, you can install them using the Software Kiosk icon on the D
Official X11 Forwarding support is through NoMachine. Please follow the document
[{Job Submission -> Interactive Jobs}](../03-Slurm-General-Documentation/interactive-jobs.md#requirements) and
[{Accessing Merlin -> NoMachine}](/merlin7/nomachine.html) for more details. However,
[{Accessing Merlin -> NoMachine}](../02-How-To-Use-Merlin/nomachine.md) for more details. However,
we provide a small recipe for enabling X11 Forwarding in Windows.
Check, if the **Xming** is installed on the Windows workstation by inspecting the

14
docs/merlin7/02-How-To-Use-Merlin/merlin_tools.md
View File

@@ -40,10 +40,10 @@ Path SpaceUsed SpaceQuota Space % FilesUsed FilesQuota Files %
└─ bio/hpce
```
{{site.data.alerts.tip}}You can change the width of the table by either passing
<code>--no-wrap</code> (to disable wrapping of the <i>Path</i>) or <code>--width N</code>
(to explicitly set some width by <code>N</code> characters).
{{site.data.alerts.end}}
!!! tip
You can change the width of the table by either passing `--no-wrap` (to
disable wrapping of the *Path*) or `--width N` (to explicitly set some
width by `N` characters).
#### Example #2: Project view
@@ -86,9 +86,9 @@ Project ID Path Owner Group
600000013 /data/project/bio/steinmetz steinmetz unx-bio_steinmetz
```
{{site.data.alerts.tip}}As above you can change the table width by pass either
<code>--no-wrap</code> or <code>--width N</code>.
{{site.data.alerts.end}}
!!! tip
As above you can change the table width by pass either `--no-wrap` or
`--width N`.
#### Example #3: Project config

15
docs/merlin7/02-How-To-Use-Merlin/software-repositories.md
View File

@@ -22,17 +22,15 @@ Key Features:
* **Broad Availability:** Commonly used software, such as OpenMPI, ANSYS, MATLAB, and other, is provided within PModules.
* **Custom Requests:** If a package, version, or feature is missing, users can contact the support team to explore feasibility for installation.
{{site.data.alerts.tip}}
For further information about **PModules** on Merlin7 please refer to the [PSI Modules](../05-Software-Support/pmodules.md) chapter.
{{site.data.alerts.end}}
!!! tip
For further information about **PModules** on Merlin7 please refer to the [PSI Modules](../05-Software-Support/pmodules.md) chapter.
### Spack Modules
Merlin7 also provides Spack modules, offering a modern and flexible package management system. Spack supports a wide variety of software packages and versions. For more information, refer to the **external [PSI Spack](https://gitea.psi.ch/HPCE/spack-psi) documentation**.
{{site.data.alerts.tip}}
For further information about **Spack** on Merlin7 please refer to the [Spack](../05-Software-Support/spack.md) chapter.
{{site.data.alerts.end}}
!!! tip
For further information about **Spack** on Merlin7 please refer to the [Spack](../05-Software-Support/spack.md) chapter.
### Cray Environment Modules
@@ -44,7 +42,6 @@ Recommendations:
* **Compiling Software:** Cray modules can be used when optimization for Cray hardware is essential.
* **General Use:** For most applications, prefer PModules, which ensure stability, backward compatibility, and long-term support.
{{site.data.alerts.tip}}
For further information about **CPE** on Merlin7 please refer to the [Cray Modules](../05-Software-Support/cray-module.env.md) chapter.
{{site.data.alerts.end}}
!!! tip
For further information about **CPE** on Merlin7 please refer to the [Cray Modules](../05-Software-Support/cray-module.env.md) chapter.

57
docs/merlin7/02-How-To-Use-Merlin/storage.md
View File

@@ -18,9 +18,11 @@ This document describes the different directories of the Merlin7 cluster.
* ***Users are responsible for backing up their own data***. Is recommended to backup the data on third party independent systems (i.e. LTS, Archive, AFS, SwitchDrive, Windows Shares, etc.).
* ***When a user leaves PSI, she or her supervisor/team are responsible to backup and move the data out from the cluster***: every few months, the storage space will be recycled for those old users who do not have an existing and valid PSI account.
{{site.data.alerts.warning}}When a user leaves PSI and their account is removed, their storage space in Merlin may be recycled.
Hence, <b>when a user leaves PSI</b>, they, their supervisor or team <b>must ensure that the data is backed up to an external storage</b>
{{site.data.alerts.end}}
!!! warning
When a user leaves PSI and their account is removed, their storage space in
Merlin may be recycled. Hence, **when a user leaves PSI**, they, their
supervisor or team **must ensure that the data is backed up to an external
storage**!
### How to check quotas
@@ -43,20 +45,22 @@ Path SpaceUsed SpaceQuota Space % FilesUsed FilesQuota Files %
└─ bio/hpce
```
{{site.data.alerts.note}}On first use you will see a message about some configuration being generated, this is expected. Don't be
surprised that it takes some time. After this using <code>merlin_quotas</code> should be faster.
{{site.data.alerts.end}}
!!! note
On first use you will see a message about some configuration being
generated, this is expected. Don't be surprised that it takes some time.
After this using `merlin_quotas` should be faster.
The output shows the quotas set and how much you are using of the quota, for each filesystem that has this set. Notice that some users will have
one or more `/data/project/...` directories showing, depending on whether you are part of a specific PSI research group or project.
The general quota constraints for the different directories are shown in the [table below](#dir_classes). Further details on how to use `merlin_quotas`
can be found on the [Tools page](/merlin7/tools.html).
can be found on the [Tools page](merlin_tools.md).
{{site.data.alerts.tip}}If you're interesting, you can retrieve the Lustre-based quota information directly by calling
<code>lfs quota -h -p $(( 100000000 + $(id -u $USER) )) /data</code> directly. Using the <code>merlin_quotas</code> command is more
convenient and shows all your relevant filesystem quotas.
{{site.data.alerts.end}}
!!! tip
If you're interesting, you can retrieve the Lustre-based quota information
directly by calling `lfs quota -h -p $(( 100000000 + $(id -u $USER) ))
/data` directly. Using the `merlin_quotas` command is more convenient and
shows all your relevant filesystem quotas.
## Merlin7 directories
@@ -70,11 +74,14 @@ Merlin7 offers the following directory classes for users:
* `/scratch`: Local *scratch* disk (only visible by the node running a job).
* `/data/scratch/shared`: Shared *scratch* disk (visible from all nodes).
{{site.data.alerts.tip}}In Lustre there is a concept called <b>grace time</b>. Filesystems have a block (amount of data) and inode (number of files) quota.
These quotas contain a soft and hard limits. Once the soft limit is reached, users can keep writing up to their hard limit quota during the <b>grace period</b>.
Once the <b>grace time</b> or hard limit are reached, users will be unable to write and will need remove data below the soft limit (or ask for a quota increase
when this is possible, see below table).
{{site.data.alerts.end}}
!!! tip
In Lustre there is a concept called **grace time**. Filesystems have a
block (amount of data) and inode (number of files) quota. These quotas
contain a soft and hard limits. Once the soft limit is reached, users can
keep writing up to their hard limit quota during the **grace period**.
Once the **grace time** or hard limit are reached, users will be unable to
write and will need remove data below the soft limit (or ask for a quota
increase when this is possible, see below table).
<a name="dir_classes"></a>Properties of the directory classes:
@@ -86,10 +93,11 @@ when this is possible, see below table).
| /data/scratch/shared | USR [512GB:2TB] | | 7d | Up to x2 when strongly justified. | Changeable when justified. | no |
| /scratch | *Undef* | *Undef* | N/A | N/A | N/A | no |
{{site.data.alerts.warning}}The use of <b>/scratch</b> and <b>/data/scratch/shared</b> areas as an extension of the quota <i>is forbidden</i>. The <b>/scratch</b> and
<b>/data/scratch/shared</b> areas <i>must not contain</i> final data. Keep in mind that <br><b><i>auto cleanup policies</i></b> in the <b>/scratch</b> and
<b>/data/scratch/shared</b> areas are applied.
{{site.data.alerts.end}}
!!! warning
The use of `/scratch` and `/data/scratch/shared` areas as an extension of
the quota *is forbidden*. The `/scratch` and `/data/scratch/shared` areas
***must not contain*** final data. Keep in mind that ***auto cleanup
policies*** in the `/scratch` and `/data/scratch/shared` areas are applied.
### User home directory
@@ -134,9 +142,10 @@ Project quotas are defined in a per Lustre project basis. Users can check the pr
lfs quota -h -p $projectid /data
```
{{site.data.alerts.warning}}Checking <b>quotas</b> for the Merlin projects is not yet possible.
In the future, a list of `projectid` will be provided, so users can check their quotas.
{{site.data.alerts.end}}
!!! warning
Checking **quotas** for the Merlin projects is not yet possible. In the
future, a list of `projectid` will be provided, so users can check their
quotas.
Directory policies:
@@ -178,7 +187,7 @@ and all tasks need to do I/O on the same temporary files.
Scratch directories policies:
* Read **[Important: Code of Conduct](/merlin7/code-of-conduct.html)** for more information about Merlin7 policies.
* Read **[Important: Code of Conduct](../01-Quick-Start-Guide/code-of-conduct.md)** for more information about Merlin7 policies.
* By default, *always* use **local** first and only use **shared** if your specific use case requires it.
* Temporary files *must be deleted at the end of the job by the user*.
* Remaining files will be deleted by the system if detected.

29
docs/merlin7/02-How-To-Use-Merlin/transfer-data.md
View File

@@ -55,12 +55,12 @@ rsync -avAHXS <src> <dst>
```bash
rsync -avAHXS ~/localdata $USER@login001.merlin7.psi.ch:/data/project/general/myproject/
```
{{site.data.alerts.tip}}
If a transfer is interrupted, just rerun the command: <code>rsync</code> will skip existing files.
{{site.data.alerts.end}}
{{site.data.alerts.warning}}
Rsync uses SSH (port 22). For large datasets, transfer speed might be limited.
{{site.data.alerts.end}}
!!! tip
If a transfer is interrupted, just rerun the command: `rsync` will skip existing files.
!!! warning
Rsync uses SSH (port 22). For large datasets, transfer speed might be limited.
### SCP
@@ -76,9 +76,8 @@ A `vsftpd` service is available on the login nodes, providing high-speed transfe
* **`service03.merlin7.psi.ch`**: Encrypted control channel only.
Use if your data can be transferred unencrypted. **Fastest** method.
{{site.data.alerts.tip}}
The <b>control channel</b> is always <b>encrypted</b>, therefore, authentication is encrypted and secured.
{{site.data.alerts.end}}
!!! tip
The **control channel** is always **encrypted**, therefore, authentication is encrypted and secured.
## UI-based Clients for Data Transfer
### WinSCP (Windows)
@@ -125,9 +124,8 @@ The service is designed to **send large files for temporary availability**, not
3. File remains available until the specified **expiration date** is reached, or the **download limit** is reached.
4. The file is **automatically deleted** after expiration.
{{site.data.alerts.warning}}
SWITCHfilesender <b>is not</b> a long-term storage or archiving solution.
{{site.data.alerts.end}}
!!! warning
SWITCHfilesender **is not** a long-term storage or archiving solution.
## PSI Data Transfer
@@ -144,9 +142,10 @@ Notice that `datatransfer.psi.ch` does not allow SSH login, only `rsync`, `scp`
Access to the PSI Data Transfer uses ***Multi factor authentication*** (MFA).
Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa).
{{site.data.alerts.tip}}Please follow the
<b><a href="https://www.psi.ch/en/photon-science-data-services/data-transfer">Official PSI Data Transfer</a></b> documentation for further instructions.
{{site.data.alerts.end}}
!!! tip
Please follow the [Official PSI Data
Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)
documentation for further instructions.
## Connecting to Merlin7 from outside PSI