HPCE/gitea-pages
HPCE/gitea-pages
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Activity

Updating

Browse Source
This commit is contained in:
caubet_m 2019-06-18 10:47:52 +02:00
parent bdda294d64
commit 2f18499dee
2 changed files with 41 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
pages/merlin6/accessing-merlin6/merlin6-directories.md
View File

@ -2,7 +2,7 @@
title: Merlin6 Data Directories
#tags:
#keywords:
last_updated: 13 June 2019
last_updated: 18 June 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/data-directories.html
@ -30,9 +30,7 @@ A summary for each directory would be:
| /scratch | *Undef* | *Undef* | N/A | N/A | no | N/A |
| /shared-scratch | *Undef* | *Undef* | N/A | N/A | no | N/A |
---
## User home directory
### User home directory
Home directories are part of the PSI NFS Central Home storage provided by AIT.
However, administration for the Merlin6 NFS homes is delegated to Merlin6 administrators.
@ -50,7 +48,7 @@ Users can check their quota by running the following command:
quota -s
```
### Home directory policy
#### Home directory policy
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the home directories for IO intensive tasks
@ -62,9 +60,7 @@ Snapshots are found in the following directory:
/psi/home/.snapshop/$username
```
---
## User data directory
### User data directory
User data directories are part of the Merlin6 storage cluster and technology is based on GPFS.
@ -81,16 +77,14 @@ Users can check their quota by running the following command:
mmlsquota -u <username> --block-size auto merlin-user
```
### User Directory policy
#### User Directory policy
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the data directories as ``scratch`` area during a job runtime.
* Use ``/scratch``, ``/shared-scratch`` for this purpose.
* No backup policy is applied for user data directories: users are responsible for backing up their data.
---
## Project data directory
### Project data directory
Project data directories are part of the Merlin6 storage cluster and technology is based on GPFS.
@ -115,16 +109,14 @@ Users can check the project quota by running the following command:
mmrepquota merlin-proj:$projectname
```
### Project Directory policy
#### Project Directory policy
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the data directories as ``scratch`` area during a job runtime.
* Use ``/scratch``, ``/shared-scratch`` for this purpose.
* No backups: users are responsible for managing the backups of their data directories.
---
## Scratch directories
### Scratch directories
There are two different types of scratch disk: **local** (``/scratch``) and **shared** (``/shared-scratch``).
Specific details of each type is described below.
@ -146,7 +138,7 @@ A summary for the scratch directories is the following:
| merlin6 | computing node | 1.3TB / NVMe | ``/scratch`` | 50TB / GPFS | ``/shared-scratch`` | ``merlin-c-[001-022,101-122,201-222`` |
| merlin6 | login node | 2.0TB / NVMe | ``/scratch`` | ``N/A`` | ``N/A`` | ``merlin-l-00[1,2]`` |
### Scratch directories policy
#### Scratch directories policy
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* By default, *always* use **local** first and only use **shared** if you specific use case needs a shared scratch area.

39
pages/merlin6/migration-from-merlin5.md
View File

@ -2,33 +2,37 @@
title: Migration From Merlin5
#tags:
#keywords:
last_updated: 13 June 2019
last_updated: 18 June 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/migrating.html
---
## Merlin5 vs Merlin6
## Directories
### Directories
### Merlin5 vs Merlin6
| Cluster | Home Directory | User Home Directory | Group Home Directory |
| ------- |:-------------------- |:-------------------- |:---------------------------------------- |
| merlin5 | /gpfs/home/_$username_ | /gpfs/data/_$username_ | /gpfs/group/_$laboratory_ |
| merlin6 | /psi/home/_$username_ | /data/user/_$username_ | /data/project/_\[general\|bio\]_/_$projectname_ |
### User/Group quota limits in Merlin6
### Quota limits in Merlin6
| Directory | Quota_Type [Soft:Hard] (Block) | Quota_Type [Soft:Hard] (Files) | Quota Change Policy: Block | Quota Change Policy: Files |
| ---------------------------------- | ------------------------------ | ------------------------------ |:--------------------------------------------- |:--------------------------------------------- |
| /psi/home/$username | USR [10GB:11GB] | *Undef* | Up to x2 when strictly justified. | N/A |
| /data/user/$username | USR [1TB:1.074TB] | USR [1M:1.1M] | Inmutable. Need a project. | Changeable when justified. |
| /data/project/bio/$projectname | GRP [1TB:1.074TB] | GRP [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. |
| /data/project/general/$projectname | GRP [1TB:1.074TB] | GRP [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. |
| /data/project/bio/$projectname | GRP+Fileset [1TB:1.074TB] | GRP+Fileset [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. |
| /data/project/general/$projectname | GRP+Fileset [1TB:1.074TB] | GRP+Fileset [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. |
where:
* **Block** is capacity size in GB and TB
* **Files** is number of files + directories in Millions (M)
* **Quota types** are the following:
* **USR**: Quota is setup individually per user name
* **GRP**: Quota is setup individually per Unix Group name
* **Fileset**: Quota is setup per project root directory.
* User data directory ``/data/user`` has a strict user block quota limit policy. If more disk space is required, 'project' must be created.
* Soft quotas can be exceeded for short periods of time. Hard quotas cannot be exceeded.
@ -38,14 +42,14 @@ where:
Merlin6 introduces the concept of a *project* directory. These are the recommended location for all scientific data.
- `/data/user` is not suitable for sharing data between users
- The Merlin5 *group* directories were a similar concept, but the association with a single organizational group made
* `/data/user` is not suitable for sharing data between users
* The Merlin5 *group* directories were a similar concept, but the association with a single organizational group made
interdepartmental sharing difficult. Projects can be shared by any PSI user.
- Projects are shared by multiple users (at a minimum they should be shared with the supervisor/PI). This decreases
* Projects are shared by multiple users (at a minimum they should be shared with the supervisor/PI). This decreases
the chance of data being orphaned by personnel changes.
- Shared projects are preferable to individual data for transparency and accountability in event of future questions
* Shared projects are preferable to individual data for transparency and accountability in event of future questions
regarding the data.
- One project member is designated as responsible. Responsibility can be transferred if needed.
* One project member is designated as responsible. Responsibility can be transferred if needed.
#### Requesting a *project*
@ -66,24 +70,28 @@ For requesting a *project* users must provide:
## Migration Schedule
### Phase 1 [June]: Pre-migration
* Users keep working on Merlin5
* Merlin5 production directories: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users may raise any problems (quota limits, unaccessible files, etc.) to merlin-admins@lists.psi.ch
* Users can start migrating data (see [Migration steps](# Migration steps))
* Users should copy their data from Merlin5 /gpfs/data to Merlin6 /data/user
* Users should copy their home from Merlin5 /gpfs/home to Merlin6 /psi/home
* Users should copy their data from Merlin5 ``/gpfs/data`` to Merlin6 ``/data/user``
* Users should copy their home from Merlin5 ``/gpfs/home`` to Merlin6 ``/psi/home``
* Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins.
### Phase 2 [July-October]: Migration to Merlin6
* Merlin6 becomes official cluster, and directories are switched to the new structure:
* Merlin6 production directories: ``'/psi/home/'``, ``'/data/user'``, ``'/data/project'``
* Merlin5 directories available in RO: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users can keep migrating their data (see [Migration steps](# Migration steps))
* Merlin5 directories available in RO in login nodes: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* In Merlin5 computing nodes, Merlin5 directories will be mounted in RW: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users must migrate their data (see [Migration steps](# Migration steps))
* ALL data must be migrated
* Job submissions by default to Merlin6. Submission to Merlin5 computing nodes possible.
* Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins.
### Phase 3 [November]: Merlin5 Decomission
* Old Merlin5 storage unmounted.
* Migrated directories reported by users will be deleted.
* Remaining Merlin5 data will be archived.
@ -94,6 +102,7 @@ For requesting a *project* users must provide:
## Migration steps
### Cleanup / Archive files
* Users must cleanup and/or archive files, according to quota limits in the storage.
* If extra space is needed, *'project'* would be needed.
* If extra files are needed, you can request for an increasement of the quota/