Add Migration Guide
All checks were successful
Build and Deploy Documentation / build-and-deploy (push) Successful in 6s
All checks were successful
Build and Deploy Documentation / build-and-deploy (push) Successful in 6s
This commit is contained in:
parent
0802f12fcf
commit
2a85a9f03c
@ -76,6 +76,8 @@ entries:
|
||||
url: /merlin7/ippl.html
|
||||
- title: Support
|
||||
folderitems:
|
||||
- title: Merlin6 to Merlin7 Migration Guide
|
||||
url: /merlin7/migrating.html
|
||||
- title: Contact
|
||||
url: /merlin7/contact.html
|
||||
|
||||
|
||||
248
pages/merlin7/99-support/migration-from-merlin6.md
Normal file
248
pages/merlin7/99-support/migration-from-merlin6.md
Normal file
@ -0,0 +1,248 @@
|
||||
---
|
||||
#tags:
|
||||
keywords: merlin6, merlin7, migration, fpsync, rsync
|
||||
#summary: ""
|
||||
sidebar: merlin7_sidebar
|
||||
last_updated: 27 May 2025
|
||||
permalink: /merlin7/migrating.html
|
||||
---
|
||||
|
||||
# Merlin6 to Merlin7 Migration Guide
|
||||
|
||||
Welcome to the official documentation for migrating your data from **Merlin6** to **Merlin7**. Please follow the instructions carefully to ensure a smooth and secure transition.
|
||||
|
||||
## 📅 Migration Schedule
|
||||
|
||||
### 🧍 Phase 1: Users without Projects — **Deadline: July 1**
|
||||
|
||||
If you **do not belong to any Merlin project**, you must complete your migration **before July 1**. This includes:
|
||||
|
||||
* Users not in any group project (`/data/projects/general`)
|
||||
* Users not in BIO, MEG, Mu3e
|
||||
* Users not part of PSI-owned private Merlin nodes (ASA, MEG, Mu3e)
|
||||
|
||||
Users are responsible for initiating and completing the migration process.
|
||||
Contact the Merlin support team [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists.psi.ch) if you need help.
|
||||
|
||||
> ⚠️ In this phase, **it's important that you don't belong to any project**.
|
||||
> Once the migration is finished, **access to Merlin6 will be no longer possible.**
|
||||
|
||||
### 👥 Phase 2: Project Members and Owners — **Start Before August 1**
|
||||
|
||||
For users in active projects:
|
||||
|
||||
* Project **owners and members will be contacted by the Merlin admins**.
|
||||
* Migration will be **scheduled individually per project**.
|
||||
* Expect contact **before August 1**.
|
||||
|
||||
> ⚠️ In this phase, **group owners and members will be also requested to be migrated.**
|
||||
|
||||
---
|
||||
|
||||
## 🗂️ Directory Structure Changes
|
||||
|
||||
### Merlin6 vs Merlin7
|
||||
|
||||
| Cluster | Home Directory | User Data Directory | Projects | Experiments |
|
||||
| ------- | :----------------- | :------------------ | -------------- | ----------------- |
|
||||
| merlin6 | /psi/home/`$USER` | /data/user/`$USER` | /data/project/ | /data/experiments |
|
||||
| merlin7 | /data/user/`$USER` | /data/user/`$USER` | /data/project/ | /data/project/ |
|
||||
|
||||
* The **home directory and user data directory have been merged** into `/data/user/$USER`.
|
||||
* The **experiments directory has been integrated into `/data/project/`**:
|
||||
|
||||
* `/data/project/general` contains general Merlin7 projects.
|
||||
* Other subdirectories are used for large-scale projects such as CLS division, Mu3e, and MeG.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Step-by-Step Migration Instructions
|
||||
|
||||
### 📋 Prerequisites and Preparation
|
||||
|
||||
Before starting the migration, make sure you:
|
||||
|
||||
* Are **registered on Merlin7**.
|
||||
|
||||
* If not yet registered, please do so at: [https://i.psi.ch/6L8](https://i.psi.ch/6L8)
|
||||
|
||||
* **Have cleaned up your data to reduce migration time and space usage**.
|
||||
* Ensure your total usage on Merlin6 is **well below the 1 TB quota**. Remember:
|
||||
|
||||
* **Merlin7 also has a 1 TB quota**, and you might already have data there.
|
||||
* If your usage exceeds this during the transfer, the process might fail.
|
||||
|
||||
**Recommended Cleanup Actions:**
|
||||
|
||||
* Remove unused files and datasets.
|
||||
* Archive large, inactive data sets.
|
||||
* Delete or clean up unused `conda` or `virtualenv` Python environments:
|
||||
|
||||
* These are often large and may not work as-is on Merlin7.
|
||||
* You can export your conda environments with:
|
||||
|
||||
```bash
|
||||
conda env export -n myenv > $HOME/myenv.yml
|
||||
```
|
||||
* Then recreate them later on Merlin7.
|
||||
|
||||
> 🧹 You can always remove more old data **after** migration — it will be copied into `~/merlin6data` and `~/merlin6home` on Merlin7.
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Step 1: Run `merlin7_migration.setup`
|
||||
|
||||
Log into **Merlin7** and run:
|
||||
|
||||
```bash
|
||||
merlin7_migration.setup
|
||||
```
|
||||
|
||||
This script will:
|
||||
|
||||
* Check that you have an account on Merlin7.
|
||||
* Configure and check that your environment is ready for transferring files via Slurm job.
|
||||
* **Create two directories:**
|
||||
|
||||
* `~/merlin6data` → copy of your old /data/user
|
||||
* `~/merlin6home` → copy of your old home
|
||||
|
||||
> ⚠️ **Important:** If `~/merlin6home` or `~/merlin6data` already exist, the script will exit.
|
||||
> **Please remove them or contact support**.
|
||||
|
||||
If there are issues, the script will:
|
||||
|
||||
* Print clear diagnostic output
|
||||
* Give you some hints to resolve the issue
|
||||
|
||||
If you are stuck, email: [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists.psi.ch)
|
||||
|
||||
---
|
||||
|
||||
## 📦 Step 2: Run `merlin7_migration.start`
|
||||
|
||||
After setup completes, start the migration by running:
|
||||
|
||||
```bash
|
||||
merlin7_migration.start
|
||||
```
|
||||
|
||||
This script will:
|
||||
|
||||
* Check the status of your quota on Merlin6.
|
||||
* Submit **SLURM batch jobs** to the **`xfer`** partition
|
||||
* Queue two jobs:
|
||||
|
||||
* `migrate_merlin6data.batch` (data dir)
|
||||
* `migrate_merlin6home.batch` (home dir)
|
||||
* Automatically track the job IDs
|
||||
* Print log file locations for the different jobs
|
||||
|
||||
> ⚠️ **Once both transfers succeed, your access to Merlin6 will be revoked.**
|
||||
> Do **not** attempt to reconnect to Merlin6 after this.
|
||||
|
||||
---
|
||||
|
||||
## 📊 Step 3: Monitor Transfer Jobs
|
||||
|
||||
To monitor your transfer jobs, run:
|
||||
|
||||
```bash
|
||||
squeue -M merlin6 -u $USER -p xfer
|
||||
```
|
||||
|
||||
Check the output to ensure your jobs are:
|
||||
|
||||
* Running (`R`) or completed (`CG` or removed from queue)
|
||||
* Not failed (`F`, `TO`, or stuck)
|
||||
|
||||
You can also check logs (as printed by the script) to verify job completion.
|
||||
|
||||
> ✅ When `/data/user/$USER` and `/psi/home/$USER` on Merlin6 are no longer accessible, migration is complete.
|
||||
|
||||
---
|
||||
|
||||
## 💡 Examples
|
||||
|
||||
### Setup the Migration
|
||||
|
||||
```bash
|
||||
merlin7_migration.setup
|
||||
```
|
||||
|
||||
*Expected output:*
|
||||
|
||||
```bash
|
||||
✅ login002.merlin7.psi.ch
|
||||
✅ `$USER` is a member of svc-cluster_merlin7
|
||||
✅ Skipping key generation
|
||||
✅ SSH key already added to agent.
|
||||
✅ SSH ID successfully copied to login00[1|2].merlin7.psi.ch.
|
||||
✅ Test successful.
|
||||
✅ /data/software/xfer_logs/caubet_m created.
|
||||
✅ ~/merlin6data directory created.
|
||||
✅ ~/merlin6home directory created.
|
||||
```
|
||||
|
||||
### Start the Migration
|
||||
|
||||
```bash
|
||||
merlin7_migration.start
|
||||
```
|
||||
|
||||
*Expected output:*
|
||||
|
||||
```bash
|
||||
(base) ❄ [caubet_m@merlin-l-001:/data/software/admin/scripts/merlin-user-tools/alps(master)]# ./merlin7_migration.start
|
||||
✅ Quota check passed.
|
||||
Used: 512 GB, 234001 files
|
||||
|
||||
###################################################
|
||||
Submitting transfer jobs to Slurm
|
||||
|
||||
Job logs can be found here:
|
||||
➡️ Directory '/data/user/caubet_m' does NOT have 000 permissions. Transfer pending, continuing...
|
||||
✅ Submitted DATA_MIGRATION job: 24688554. Sleeping 3 seconds...
|
||||
- /data/user transfer logs:
|
||||
- /data/software/xfer_logs/caubet_m/data-24688554.out
|
||||
- /data/software/xfer_logs/caubet_m/data-24688554.err
|
||||
➡️ Directory '/psi/home/caubet_m' does NOT have 000 permissions. Transfer pending, continuing...
|
||||
✅ Submitted HOME_MIGRATION job with dependency on 24688554: 24688555. Sleeping 3 seconds...
|
||||
- /psi/home transfer logs:
|
||||
- /data/software/xfer_logs/caubet_m/home-24688555.out
|
||||
- /data/software/xfer_logs/caubet_m/home-24688555.err
|
||||
|
||||
✅ You can start manually a monitoring window with:
|
||||
tmux new-session -d -s "xfersession" "watch 'squeue -M merlin6 -u caubet_m -p xfer'"
|
||||
tmux attach -t "xfersession"
|
||||
|
||||
✅ FINISHED
|
||||
```
|
||||
|
||||
### Monitor Progress
|
||||
|
||||
```bash
|
||||
squeue -M merlin6 -u $USER -p xfer
|
||||
```
|
||||
|
||||
*Output:*
|
||||
|
||||
```bash
|
||||
$ squeue -M merlin6 -u $USER -p xfer
|
||||
CLUSTER: merlin6
|
||||
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
|
||||
24688581 xfer HOME_MIG caubet_m PD 0:00 1 (Dependency)
|
||||
24688580 xfer DATA_MIG caubet_m R 0:22 1 merlin-c-017
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📬 Need Help?
|
||||
|
||||
If something doesn't work:
|
||||
|
||||
* Re-run the scripts and check the logs carefully.
|
||||
* Use `less`, `cat`, or `tail -f` to view your job logs.
|
||||
* Contact the Merlin support team: 📧 [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists.psi.ch)
|
||||
|
||||
> We are here to help you migrate safely and efficiently.
|
||||
Loading…
x
Reference in New Issue
Block a user