HPCE/gitea-pages
HPCE/gitea-pages
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Activity

Updating

Browse Source
This commit is contained in:
caubet_m 2019-06-18 10:47:52 +02:00
parent bdda294d64
commit 2f18499dee
2 changed files with 41 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
pages/merlin6/accessing-merlin6/merlin6-directories.md
View File

@ -2,7 +2,7 @@
title: Merlin6 Data Directories title: Merlin6 Data Directories
#tags: #tags:
#keywords: #keywords:
last_updated: 13 June 2019 last_updated: 18 June 2019
#summary: "" #summary: ""
sidebar: merlin6_sidebar sidebar: merlin6_sidebar
permalink: /merlin6/data-directories.html permalink: /merlin6/data-directories.html
@ -30,9 +30,7 @@ A summary for each directory would be:
| /scratch | *Undef* | *Undef* | N/A | N/A | no | N/A | | /scratch | *Undef* | *Undef* | N/A | N/A | no | N/A |
| /shared-scratch | *Undef* | *Undef* | N/A | N/A | no | N/A | | /shared-scratch | *Undef* | *Undef* | N/A | N/A | no | N/A |
--- ### User home directory
## User home directory
Home directories are part of the PSI NFS Central Home storage provided by AIT. Home directories are part of the PSI NFS Central Home storage provided by AIT.
However, administration for the Merlin6 NFS homes is delegated to Merlin6 administrators. However, administration for the Merlin6 NFS homes is delegated to Merlin6 administrators.
@ -50,7 +48,7 @@ Users can check their quota by running the following command:
quota -s quota -s
``` ```
### Home directory policy #### Home directory policy
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies. * Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the home directories for IO intensive tasks * Is **forbidden** to use the home directories for IO intensive tasks
@ -62,9 +60,7 @@ Snapshots are found in the following directory:
/psi/home/.snapshop/$username /psi/home/.snapshop/$username
``` ```
--- ### User data directory
## User data directory
User data directories are part of the Merlin6 storage cluster and technology is based on GPFS. User data directories are part of the Merlin6 storage cluster and technology is based on GPFS.
@ -81,16 +77,14 @@ Users can check their quota by running the following command:
mmlsquota -u <username> --block-size auto merlin-user mmlsquota -u <username> --block-size auto merlin-user
``` ```
### User Directory policy #### User Directory policy
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies. * Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the data directories as ``scratch`` area during a job runtime. * Is **forbidden** to use the data directories as ``scratch`` area during a job runtime.
* Use ``/scratch``, ``/shared-scratch`` for this purpose. * Use ``/scratch``, ``/shared-scratch`` for this purpose.
* No backup policy is applied for user data directories: users are responsible for backing up their data. * No backup policy is applied for user data directories: users are responsible for backing up their data.
--- ### Project data directory
## Project data directory
Project data directories are part of the Merlin6 storage cluster and technology is based on GPFS. Project data directories are part of the Merlin6 storage cluster and technology is based on GPFS.
@ -115,16 +109,14 @@ Users can check the project quota by running the following command:
mmrepquota merlin-proj:$projectname mmrepquota merlin-proj:$projectname
``` ```
### Project Directory policy #### Project Directory policy
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies. * Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the data directories as ``scratch`` area during a job runtime. * Is **forbidden** to use the data directories as ``scratch`` area during a job runtime.
* Use ``/scratch``, ``/shared-scratch`` for this purpose. * Use ``/scratch``, ``/shared-scratch`` for this purpose.
* No backups: users are responsible for managing the backups of their data directories. * No backups: users are responsible for managing the backups of their data directories.
--- ### Scratch directories
## Scratch directories
There are two different types of scratch disk: **local** (``/scratch``) and **shared** (``/shared-scratch``). There are two different types of scratch disk: **local** (``/scratch``) and **shared** (``/shared-scratch``).
Specific details of each type is described below. Specific details of each type is described below.
@ -146,7 +138,7 @@ A summary for the scratch directories is the following:
| merlin6 | computing node | 1.3TB / NVMe | ``/scratch`` | 50TB / GPFS | ``/shared-scratch`` | ``merlin-c-[001-022,101-122,201-222`` | | merlin6 | computing node | 1.3TB / NVMe | ``/scratch`` | 50TB / GPFS | ``/shared-scratch`` | ``merlin-c-[001-022,101-122,201-222`` |
| merlin6 | login node | 2.0TB / NVMe | ``/scratch`` | ``N/A`` | ``N/A`` | ``merlin-l-00[1,2]`` | | merlin6 | login node | 2.0TB / NVMe | ``/scratch`` | ``N/A`` | ``N/A`` | ``merlin-l-00[1,2]`` |
### Scratch directories policy #### Scratch directories policy
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies. * Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* By default, *always* use **local** first and only use **shared** if you specific use case needs a shared scratch area. * By default, *always* use **local** first and only use **shared** if you specific use case needs a shared scratch area.

39
pages/merlin6/migration-from-merlin5.md
View File

@ -2,33 +2,37 @@
title: Migration From Merlin5 title: Migration From Merlin5
#tags: #tags:
#keywords: #keywords:
last_updated: 13 June 2019 last_updated: 18 June 2019
#summary: "" #summary: ""
sidebar: merlin6_sidebar sidebar: merlin6_sidebar
permalink: /merlin6/migrating.html permalink: /merlin6/migrating.html
--- ---
## Merlin5 vs Merlin6 ## Directories
### Directories ### Merlin5 vs Merlin6
| Cluster | Home Directory | User Home Directory | Group Home Directory | | Cluster | Home Directory | User Home Directory | Group Home Directory |
| ------- |:-------------------- |:-------------------- |:---------------------------------------- | | ------- |:-------------------- |:-------------------- |:---------------------------------------- |
| merlin5 | /gpfs/home/_$username_ | /gpfs/data/_$username_ | /gpfs/group/_$laboratory_ | | merlin5 | /gpfs/home/_$username_ | /gpfs/data/_$username_ | /gpfs/group/_$laboratory_ |
| merlin6 | /psi/home/_$username_ | /data/user/_$username_ | /data/project/_\[general\|bio\]_/_$projectname_ | | merlin6 | /psi/home/_$username_ | /data/user/_$username_ | /data/project/_\[general\|bio\]_/_$projectname_ |
### User/Group quota limits in Merlin6 ### Quota limits in Merlin6
| Directory | Quota_Type [Soft:Hard] (Block) | Quota_Type [Soft:Hard] (Files) | Quota Change Policy: Block | Quota Change Policy: Files | | Directory | Quota_Type [Soft:Hard] (Block) | Quota_Type [Soft:Hard] (Files) | Quota Change Policy: Block | Quota Change Policy: Files |
| ---------------------------------- | ------------------------------ | ------------------------------ |:--------------------------------------------- |:--------------------------------------------- | | ---------------------------------- | ------------------------------ | ------------------------------ |:--------------------------------------------- |:--------------------------------------------- |
| /psi/home/$username | USR [10GB:11GB] | *Undef* | Up to x2 when strictly justified. | N/A | | /psi/home/$username | USR [10GB:11GB] | *Undef* | Up to x2 when strictly justified. | N/A |
| /data/user/$username | USR [1TB:1.074TB] | USR [1M:1.1M] | Inmutable. Need a project. | Changeable when justified. | | /data/user/$username | USR [1TB:1.074TB] | USR [1M:1.1M] | Inmutable. Need a project. | Changeable when justified. |
| /data/project/bio/$projectname | GRP [1TB:1.074TB] | GRP [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. | | /data/project/bio/$projectname | GRP+Fileset [1TB:1.074TB] | GRP+Fileset [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. |
| /data/project/general/$projectname | GRP [1TB:1.074TB] | GRP [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. | | /data/project/general/$projectname | GRP+Fileset [1TB:1.074TB] | GRP+Fileset [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. |
where: where:
* **Block** is capacity size in GB and TB * **Block** is capacity size in GB and TB
* **Files** is number of files + directories in Millions (M) * **Files** is number of files + directories in Millions (M)
* **Quota types** are the following:
* **USR**: Quota is setup individually per user name
* **GRP**: Quota is setup individually per Unix Group name
* **Fileset**: Quota is setup per project root directory.
* User data directory ``/data/user`` has a strict user block quota limit policy. If more disk space is required, 'project' must be created. * User data directory ``/data/user`` has a strict user block quota limit policy. If more disk space is required, 'project' must be created.
* Soft quotas can be exceeded for short periods of time. Hard quotas cannot be exceeded. * Soft quotas can be exceeded for short periods of time. Hard quotas cannot be exceeded.
@ -38,14 +42,14 @@ where:
Merlin6 introduces the concept of a *project* directory. These are the recommended location for all scientific data. Merlin6 introduces the concept of a *project* directory. These are the recommended location for all scientific data.
- `/data/user` is not suitable for sharing data between users * `/data/user` is not suitable for sharing data between users
- The Merlin5 *group* directories were a similar concept, but the association with a single organizational group made * The Merlin5 *group* directories were a similar concept, but the association with a single organizational group made
interdepartmental sharing difficult. Projects can be shared by any PSI user. interdepartmental sharing difficult. Projects can be shared by any PSI user.
- Projects are shared by multiple users (at a minimum they should be shared with the supervisor/PI). This decreases * Projects are shared by multiple users (at a minimum they should be shared with the supervisor/PI). This decreases
the chance of data being orphaned by personnel changes. the chance of data being orphaned by personnel changes.
- Shared projects are preferable to individual data for transparency and accountability in event of future questions * Shared projects are preferable to individual data for transparency and accountability in event of future questions
regarding the data. regarding the data.
- One project member is designated as responsible. Responsibility can be transferred if needed. * One project member is designated as responsible. Responsibility can be transferred if needed.
#### Requesting a *project* #### Requesting a *project*
@ -66,24 +70,28 @@ For requesting a *project* users must provide:
## Migration Schedule ## Migration Schedule
### Phase 1 [June]: Pre-migration ### Phase 1 [June]: Pre-migration
* Users keep working on Merlin5 * Users keep working on Merlin5
* Merlin5 production directories: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'`` * Merlin5 production directories: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users may raise any problems (quota limits, unaccessible files, etc.) to merlin-admins@lists.psi.ch * Users may raise any problems (quota limits, unaccessible files, etc.) to merlin-admins@lists.psi.ch
* Users can start migrating data (see [Migration steps](# Migration steps)) * Users can start migrating data (see [Migration steps](# Migration steps))
* Users should copy their data from Merlin5 /gpfs/data to Merlin6 /data/user * Users should copy their data from Merlin5 ``/gpfs/data`` to Merlin6 ``/data/user``
* Users should copy their home from Merlin5 /gpfs/home to Merlin6 /psi/home * Users should copy their home from Merlin5 ``/gpfs/home`` to Merlin6 ``/psi/home``
* Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins. * Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins.
### Phase 2 [July-October]: Migration to Merlin6 ### Phase 2 [July-October]: Migration to Merlin6
* Merlin6 becomes official cluster, and directories are switched to the new structure: * Merlin6 becomes official cluster, and directories are switched to the new structure:
* Merlin6 production directories: ``'/psi/home/'``, ``'/data/user'``, ``'/data/project'`` * Merlin6 production directories: ``'/psi/home/'``, ``'/data/user'``, ``'/data/project'``
* Merlin5 directories available in RO: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'`` * Merlin5 directories available in RO in login nodes: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users can keep migrating their data (see [Migration steps](# Migration steps)) * In Merlin5 computing nodes, Merlin5 directories will be mounted in RW: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users must migrate their data (see [Migration steps](# Migration steps))
* ALL data must be migrated * ALL data must be migrated
* Job submissions by default to Merlin6. Submission to Merlin5 computing nodes possible. * Job submissions by default to Merlin6. Submission to Merlin5 computing nodes possible.
* Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins. * Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins.
### Phase 3 [November]: Merlin5 Decomission ### Phase 3 [November]: Merlin5 Decomission
* Old Merlin5 storage unmounted. * Old Merlin5 storage unmounted.
* Migrated directories reported by users will be deleted. * Migrated directories reported by users will be deleted.
* Remaining Merlin5 data will be archived. * Remaining Merlin5 data will be archived.
@ -94,6 +102,7 @@ For requesting a *project* users must provide:
## Migration steps ## Migration steps
### Cleanup / Archive files ### Cleanup / Archive files
* Users must cleanup and/or archive files, according to quota limits in the storage. * Users must cleanup and/or archive files, according to quota limits in the storage.
* If extra space is needed, *'project'* would be needed. * If extra space is needed, *'project'* would be needed.
* If extra files are needed, you can request for an increasement of the quota/ * If extra files are needed, you can request for an increasement of the quota/