diff --git a/_data/sidebars/merlin7_sidebar.yml b/_data/sidebars/merlin7_sidebar.yml index d6a0971..3ff81db 100644 --- a/_data/sidebars/merlin7_sidebar.yml +++ b/_data/sidebars/merlin7_sidebar.yml @@ -76,6 +76,8 @@ entries: url: /merlin7/ippl.html - title: Support folderitems: + - title: Merlin6 to Merlin7 Migration Guide + url: /merlin7/migrating.html - title: Contact url: /merlin7/contact.html diff --git a/pages/merlin7/99-support/migration-from-merlin6.md b/pages/merlin7/99-support/migration-from-merlin6.md new file mode 100644 index 0000000..ab995af --- /dev/null +++ b/pages/merlin7/99-support/migration-from-merlin6.md @@ -0,0 +1,248 @@ +--- +#tags: +keywords: merlin6, merlin7, migration, fpsync, rsync +#summary: "" +sidebar: merlin7_sidebar +last_updated: 27 May 2025 +permalink: /merlin7/migrating.html +--- + +# Merlin6 to Merlin7 Migration Guide + +Welcome to the official documentation for migrating your data from **Merlin6** to **Merlin7**. Please follow the instructions carefully to ensure a smooth and secure transition. + +## ๐Ÿ“… Migration Schedule + +### ๐Ÿง Phase 1: Users without Projects โ€” **Deadline: July 1** + +If you **do not belong to any Merlin project**, you must complete your migration **before July 1**. This includes: + +* Users not in any group project (`/data/projects/general`) +* Users not in BIO, MEG, Mu3e +* Users not part of PSI-owned private Merlin nodes (ASA, MEG, Mu3e) + +Users are responsible for initiating and completing the migration process. +Contact the Merlin support team [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists.psi.ch) if you need help. + +> โš ๏ธ In this phase, **it's important that you don't belong to any project**. +> Once the migration is finished, **access to Merlin6 will be no longer possible.** + +### ๐Ÿ‘ฅ Phase 2: Project Members and Owners โ€” **Start Before August 1** + +For users in active projects: + +* Project **owners and members will be contacted by the Merlin admins**. +* Migration will be **scheduled individually per project**. +* Expect contact **before August 1**. + +> โš ๏ธ In this phase, **group owners and members will be also requested to be migrated.** + +--- + +## ๐Ÿ—‚๏ธ Directory Structure Changes + +### Merlin6 vs Merlin7 + +| Cluster | Home Directory | User Data Directory | Projects | Experiments | +| ------- | :----------------- | :------------------ | -------------- | ----------------- | +| merlin6 | /psi/home/`$USER` | /data/user/`$USER` | /data/project/ | /data/experiments | +| merlin7 | /data/user/`$USER` | /data/user/`$USER` | /data/project/ | /data/project/ | + +* The **home directory and user data directory have been merged** into `/data/user/$USER`. +* The **experiments directory has been integrated into `/data/project/`**: + + * `/data/project/general` contains general Merlin7 projects. + * Other subdirectories are used for large-scale projects such as CLS division, Mu3e, and MeG. + +--- + +## โœ… Step-by-Step Migration Instructions + +### ๐Ÿ“‹ Prerequisites and Preparation + +Before starting the migration, make sure you: + +* Are **registered on Merlin7**. + + * If not yet registered, please do so at: [https://i.psi.ch/6L8](https://i.psi.ch/6L8) + +* **Have cleaned up your data to reduce migration time and space usage**. +* Ensure your total usage on Merlin6 is **well below the 1โ€ฏTB quota**. Remember: + + * **Merlin7 also has a 1โ€ฏTB quota**, and you might already have data there. + * If your usage exceeds this during the transfer, the process might fail. + +**Recommended Cleanup Actions:** + +* Remove unused files and datasets. +* Archive large, inactive data sets. +* Delete or clean up unused `conda` or `virtualenv` Python environments: + + * These are often large and may not work as-is on Merlin7. + * You can export your conda environments with: + + ```bash + conda env export -n myenv > $HOME/myenv.yml + ``` + * Then recreate them later on Merlin7. + +> ๐Ÿงน You can always remove more old data **after** migration โ€” it will be copied into `~/merlin6data` and `~/merlin6home` on Merlin7. + +--- + +## โš™๏ธ Step 1: Run `merlin7_migration.setup` + +Log into **Merlin7** and run: + +```bash +merlin7_migration.setup +``` + +This script will: + +* Check that you have an account on Merlin7. +* Configure and check that your environment is ready for transferring files via Slurm job. +* **Create two directories:** + + * `~/merlin6data` โ†’ copy of your old /data/user + * `~/merlin6home` โ†’ copy of your old home + +> โš ๏ธ **Important:** If `~/merlin6home` or `~/merlin6data` already exist, the script will exit. +> **Please remove them or contact support**. + +If there are issues, the script will: + +* Print clear diagnostic output +* Give you some hints to resolve the issue + +If you are stuck, email: [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists.psi.ch) + +--- + +## ๐Ÿ“ฆ Step 2: Run `merlin7_migration.start` + +After setup completes, start the migration by running: + +```bash +merlin7_migration.start +``` + +This script will: + +* Check the status of your quota on Merlin6. +* Submit **SLURM batch jobs** to the **`xfer`** partition +* Queue two jobs: + + * `migrate_merlin6data.batch` (data dir) + * `migrate_merlin6home.batch` (home dir) +* Automatically track the job IDs +* Print log file locations for the different jobs + +> โš ๏ธ **Once both transfers succeed, your access to Merlin6 will be revoked.** +> Do **not** attempt to reconnect to Merlin6 after this. + +--- + +## ๐Ÿ“Š Step 3: Monitor Transfer Jobs + +To monitor your transfer jobs, run: + +```bash +squeue -M merlin6 -u $USER -p xfer +``` + +Check the output to ensure your jobs are: + +* Running (`R`) or completed (`CG` or removed from queue) +* Not failed (`F`, `TO`, or stuck) + +You can also check logs (as printed by the script) to verify job completion. + +> โœ… When `/data/user/$USER` and `/psi/home/$USER` on Merlin6 are no longer accessible, migration is complete. + +--- + +## ๐Ÿ’ก Examples + +### Setup the Migration + +```bash +merlin7_migration.setup +``` + +*Expected output:* + +```bash +โœ… login002.merlin7.psi.ch +โœ… `$USER` is a member of svc-cluster_merlin7 +โœ… Skipping key generation +โœ… SSH key already added to agent. +โœ… SSH ID successfully copied to login00[1|2].merlin7.psi.ch. +โœ… Test successful. +โœ… /data/software/xfer_logs/caubet_m created. +โœ… ~/merlin6data directory created. +โœ… ~/merlin6home directory created. +``` + +### Start the Migration + +```bash +merlin7_migration.start +``` + +*Expected output:* + +```bash +(base) โ„ [caubet_m@merlin-l-001:/data/software/admin/scripts/merlin-user-tools/alps(master)]# ./merlin7_migration.start +โœ… Quota check passed. +Used: 512 GB, 234001 files + +################################################### +Submitting transfer jobs to Slurm + + Job logs can be found here: +โžก๏ธ Directory '/data/user/caubet_m' does NOT have 000 permissions. Transfer pending, continuing... +โœ… Submitted DATA_MIGRATION job: 24688554. Sleeping 3 seconds... + - /data/user transfer logs: + - /data/software/xfer_logs/caubet_m/data-24688554.out + - /data/software/xfer_logs/caubet_m/data-24688554.err +โžก๏ธ Directory '/psi/home/caubet_m' does NOT have 000 permissions. Transfer pending, continuing... +โœ… Submitted HOME_MIGRATION job with dependency on 24688554: 24688555. Sleeping 3 seconds... + - /psi/home transfer logs: + - /data/software/xfer_logs/caubet_m/home-24688555.out + - /data/software/xfer_logs/caubet_m/home-24688555.err + +โœ… You can start manually a monitoring window with: + tmux new-session -d -s "xfersession" "watch 'squeue -M merlin6 -u caubet_m -p xfer'" + tmux attach -t "xfersession" + +โœ… FINISHED +``` + +### Monitor Progress + +```bash +squeue -M merlin6 -u $USER -p xfer +``` + +*Output:* + +```bash +$ squeue -M merlin6 -u $USER -p xfer +CLUSTER: merlin6 + JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) + 24688581 xfer HOME_MIG caubet_m PD 0:00 1 (Dependency) + 24688580 xfer DATA_MIG caubet_m R 0:22 1 merlin-c-017 +``` + +--- + +## ๐Ÿ“ฌ Need Help? + +If something doesn't work: + +* Re-run the scripts and check the logs carefully. +* Use `less`, `cat`, or `tail -f` to view your job logs. +* Contact the Merlin support team: ๐Ÿ“ง [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists.psi.ch) + +> We are here to help you migrate safely and efficiently.