HPCE/gitea-pages
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Activity

Migrating merlin6 user guide from jekyll-example1

Browse Source 
From lsm-hpce/jekyll-example1 1eada07
This commit is contained in:
Spencer Bliven
2019-06-14 15:38:22 +02:00
parent 7c6f7b177d
commit ebff53c62c
19 changed files with 598 additions and 763 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
pages/merlin6/migration-from-merlin5.md
View File

@ -1,20 +1,11 @@
---
layout: default
title: Migration From Merlin5
parent: Merlin6 User Guide
nav_order: 7
---
# Migration From Merlin5
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
#tags:
#keywords:
last_updated: 13 June 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/migrating.html
---
## Merlin5 vs Merlin6
@ -38,7 +29,7 @@ nav_order: 7
where:
* **Block** is capacity size in GB and TB
* **Files** is number of files + directories in Millions (M)
* User data directorry ``/data/user`` has a strict user block quota limit policy. If more disk space is required, 'project' must be created.
* User data directorry ``/data/user`` has a strict user block quota limit policy. If more disk space is required, 'project' must be created.
### Project directory
@ -46,26 +37,26 @@ where:
In Merlin5 the concept *project* did not exist. A similar concept (*group*) was existing and was mostly focused for BIO experiments.
Quite often different users are working in *a similar* / *the same* project. Data was shared in different ways,
such like by allowing other users to access private data, or by having duplicates on each user directory needing access to that data.
Quite often different users are working in *a similar* / *the same* project. Data was shared in different ways,
such like by allowing other users to access private data, or by having duplicates on each user directory needing access to that data.
This makes the storage usage unefficient and insecure.
Also, there is another problem related to that: when a user leaves, we have plenty of data which needs to be kept and nobody becomes
responsible for that. In addition, after several months user is unregistered from PSI and we end up with orphaned data which needs to
Also, there is another problem related to that: when a user leaves, we have plenty of data which needs to be kept and nobody becomes
responsible for that. In addition, after several months user is unregistered from PSI and we end up with orphaned data which needs to
be kept, but we sometimes loose track of the user.
With that, we want to restrict the usage of individual data and bet for project (shared) data. There will be one main responsible for
this project, but if for some reason this person leaves, responsible can be somebody else (successor if exists, supervisor, or in the
With that, we want to restrict the usage of individual data and bet for project (shared) data. There will be one main responsible for
this project, but if for some reason this person leaves, responsible can be somebody else (successor if exists, supervisor, or in the
worst case, the admin).
#### Requesting a *project*
For requesting a *project* users must provide:
* Define a *'project'* directory name. This must be unique.
* Define a *'project'* directory name. This must be unique.
* Have an existing *project* **Unix Group**.
* This can be requested through [PSI Service Now](https://psi.service-now.com/psisp)
* Unix group must start with *``unx-``*
* Unix group must start with *``unx-``*
* This Unix Group will be the default group for the *'project'*
* Define a project main responsible and supervisor
* Define and justify quota requirements:
@ -78,7 +69,7 @@ For requesting a *project* users must provide:
### Phase 1 [June]: Pre-migration
* Users keep working on Merlin5
* Merlin5 production directories: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Merlin5 production directories: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users may raise any problems (quota limits, unaccessible files, etc.) to merlin-admins@lists.psi.ch
* Users can start migrating data (see [Migration steps](# Migration steps))
* Users should copy their data from Merlin5 /gpfs/data to Merlin6 /data/user
@ -87,11 +78,11 @@ For requesting a *project* users must provide:
### Phase 2 [July-October]: Migration to Merlin6
* Merlin6 becomes official cluster, and directories are switched to the new structure:
* Merlin6 production directories: ``'/psi/home/'``, ``'/data/user'``, ``'/data/project'``
* Merlin5 directories available in RO: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Merlin6 production directories: ``'/psi/home/'``, ``'/data/user'``, ``'/data/project'``
* Merlin5 directories available in RO: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users can keep migrating their data (see [Migration steps](# Migration steps))
* ALL data must be migrated
* Job submissions by default to Merlin6. Submission to Merlin5 computing nodes possible.
* Job submissions by default to Merlin6. Submission to Merlin5 computing nodes possible.
* Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins.
### Phase 3 [November]: Merlin5 Decomission
@ -128,7 +119,7 @@ This can take several hours or days:
### Step 2: Mirroring
Once first migration is done, a second ``rsync`` should be ran. This is done with ``--delete``. With this option ``rsync`` will
behave in a way where it will delete from the destination all files that were removed in the source, but also will propagate
behave in a way where it will delete from the destination all files that were removed in the source, but also will propagate
new files from the source to the destination.
```bash