initial formatting changes complete
This commit is contained in:
@@ -15,8 +15,8 @@ Welcome to the official documentation for migrating experiment data from **MEG**
|
||||
| merlin7 | /data/user/`$USER` | /data/user/`$USER` | /data/project/meg | |
|
||||
|
||||
* The **Merlin6 home and user data directores have been merged** into the single new home directory `/data/user/$USER` on Merlin7.
|
||||
* This is the same for the home directory in the meg cluster, which has to be merged into `/data/user/$USER` on Merlin7.
|
||||
* Users are responsible for moving the data.
|
||||
* This is the same for the home directory in the meg cluster, which has to be merged into `/data/user/$USER` on Merlin7.
|
||||
* Users are responsible for moving the data.
|
||||
* The **experiment directory has been integrated into `/data/project/meg`**.
|
||||
|
||||
### Recommended Cleanup Actions
|
||||
@@ -38,16 +38,17 @@ A `experiment_migration.setup` migration script must be executed from **any MeG
|
||||
|
||||
* The script **must be executed after every reboot** of the destination nodes.
|
||||
* **Reason:** On Merlin7, the home directory for the `root` user resides on ephemeral storage (no physical disk).
|
||||
|
||||
After a reboot, this directory is cleaned, so **SSH keys need to be redeployed** before running the migration again.
|
||||
|
||||
#### When using a PSI Active Directory (AD) account
|
||||
|
||||
* Applicable accounts include, for example:
|
||||
* `gac-meg2_data`
|
||||
* `gac-meg2`
|
||||
* `gac-meg2_data`
|
||||
* `gac-meg2`
|
||||
* The script only needs to be executed **once**, provided that:
|
||||
* The home directory for the AD account is located on a shared storage area.
|
||||
* This shared storage is accessible from the node executing the transfer.
|
||||
* The home directory for the AD account is located on a shared storage area.
|
||||
* This shared storage is accessible from the node executing the transfer.
|
||||
* **Reason:** On Merlin7, these accounts have their home directories on persistent shared storage, so the SSH keys remain available across reboots.
|
||||
|
||||
To run it:
|
||||
@@ -71,25 +72,29 @@ If you are stuck, email: [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists
|
||||
### Migration Procedure
|
||||
|
||||
1. **Run an initial sync**, ideally within a `tmux` session
|
||||
* This copies the bulk of the data from MeG to Merlin7.
|
||||
* **IMPORTANT: Do not modify the destination directories**
|
||||
* Please, before starting the transfer ensure that:
|
||||
* The source and destination directories are correct.
|
||||
* The destination directories exist.
|
||||
* This copies the bulk of the data from MeG to Merlin7.
|
||||
* **IMPORTANT: Do not modify the destination directories**
|
||||
* Please, before starting the transfer ensure that:
|
||||
* The source and destination directories are correct.
|
||||
* The destination directories exist.
|
||||
|
||||
2. **Run additional syncs if needed**
|
||||
* Subsequent syncs can be executed to transfer changes.
|
||||
* Ensure that **only one sync for the same directory runs at a time**.
|
||||
* Multiple syncs are often required since the first one may take several hours or even days.
|
||||
* Subsequent syncs can be executed to transfer changes.
|
||||
* Ensure that **only one sync for the same directory runs at a time**.
|
||||
* Multiple syncs are often required since the first one may take several hours or even days.
|
||||
|
||||
3. Schedule a date for the final migration:
|
||||
* Any activity must be stopped on the source directory.
|
||||
* In the same way, no activity must be done on the destination until the migration is complete.
|
||||
* Any activity must be stopped on the source directory.
|
||||
* In the same way, no activity must be done on the destination until the migration is complete.
|
||||
|
||||
4. **Perform a final sync with the `-E` option** (if it applies)
|
||||
* Use `-E` **only if you need to delete files on the destination that were removed from the source.**
|
||||
* This ensures the destination becomes an exact mirror of the source.
|
||||
* **Never use `-E` after the destination has gone into production**, as it will delete new data created there.
|
||||
* Use `-E` **only if you need to delete files on the destination that were removed from the source.**
|
||||
* This ensures the destination becomes an exact mirror of the source.
|
||||
* **Never use `-E` after the destination has gone into production**, as it will delete new data created there.
|
||||
|
||||
5. Disable access on the source folder.
|
||||
6. Enable access on the destination folder.
|
||||
* At this point, **no new syncs have to be performed.**
|
||||
* At this point, **no new syncs have to be performed.**
|
||||
|
||||
!!! note "Important"
|
||||
|
||||
@@ -160,9 +165,9 @@ The following example demonstrates how to migrate the **entire `online`** direct
|
||||
From: /meg/data1/online
|
||||
To: login001.merlin7.psi.ch:/data/project/meg/data1/online
|
||||
Threads: 10 | Split: 20000 files | Max size: 100G
|
||||
RunID:
|
||||
RunID:
|
||||
|
||||
Please confirm to start (y/N):
|
||||
Please confirm to start (y/N):
|
||||
❌ Transfer cancelled by user.
|
||||
```
|
||||
|
||||
@@ -181,7 +186,7 @@ The following example demonstrates how to migrate **only a subdirectory**. In th
|
||||
From: /meg/data1/shared/subprojects/meg1
|
||||
To: login002.merlin7.psi.ch:/data/project/meg/data1/shared/subprojects/meg1
|
||||
Threads: 10 | Split: 20000 files | Max size: 100G
|
||||
RunID:
|
||||
RunID:
|
||||
|
||||
Please confirm to start (y/N): N
|
||||
❌ Transfer cancelled by user.
|
||||
@@ -196,5 +201,5 @@ This command initiates the migration of the directory, by creating the destinati
|
||||
```
|
||||
|
||||
* Runs FPSYNC with 10 threads and N parts of max 20000 files or 100G files:
|
||||
* Source: `/meg/data1/shared/subprojects/meg1`
|
||||
* Destination: `login002.merlin7.psi.ch:/data/project/meg/data1/shared/subprojects/meg1`
|
||||
* Source: `/meg/data1/shared/subprojects/meg1`
|
||||
* Destination: `login002.merlin7.psi.ch:/data/project/meg/data1/shared/subprojects/meg1`
|
||||
|
||||
Reference in New Issue
Block a user