refactor CSCS and Meg content
This commit is contained in:
@@ -1,18 +0,0 @@
|
||||
---
|
||||
title: Introduction
|
||||
#tags:
|
||||
keywords: introduction, home, welcome, architecture, design
|
||||
last_updated: 07 September 2022
|
||||
#summary: "MeG cluster overview"
|
||||
sidebar: meg_sidebar
|
||||
permalink: /meg/introduction.html
|
||||
redirect_from:
|
||||
- /meg
|
||||
- /meg/index.html
|
||||
---
|
||||
|
||||
## The Meg local HPC cluster
|
||||
|
||||
"The MEG II collaboration includes almost 70 physicists from research institutions from five countries. Researchers and technicians from PSI have played a leading role, particularly with providing the high-quality beam, technical support in the detector integration, and in the design, construction, and operation of the detector readout electronics." [Source](https://www.psi.ch/en/cnm/news/in-search-of-new-physics-new-result-from-the-meg-ii-collaboration)
|
||||
|
||||
The MEG data analysis cluster is a cluster tightly coupled to Merlin and dedicated to the analysis of data from the MEG experiment. Operated for the Muon Physics group.
|
||||
@@ -1,42 +1,39 @@
|
||||
---
|
||||
title: Contact
|
||||
#tags:
|
||||
keywords: contact, support, snow, service now, mailing list, mailing, email, mail, meg-admins@lists.psi.ch, merlin users
|
||||
last_updated: 15. Jan 2025
|
||||
#summary: ""
|
||||
sidebar: meg_sidebar
|
||||
permalink: /meg/contact.html
|
||||
---
|
||||
|
||||
## Support
|
||||
# Support
|
||||
|
||||
Support can be asked through:
|
||||
|
||||
* [PSI Service Now](https://psi.service-now.com/psisp)
|
||||
* E-Mail: <meg-admins@lists.psi.ch>
|
||||
|
||||
Basic contact information is also displayed on every shell login to the system using the *Message of the Day* mechanism.
|
||||
Basic contact information is also displayed on every shell login to the system
|
||||
using the *Message of the Day* mechanism.
|
||||
|
||||
|
||||
### PSI Service Now
|
||||
## PSI Service Now
|
||||
|
||||
**[PSI Service Now](https://psi.service-now.com/psisp)**: is the official PSI tool for opening incident requests. However, contact via email (see below) is preferred.
|
||||
* PSI HelpDesk will redirect the incident to the corresponding department, or
|
||||
* you can always assign it directly by checking the box `I know which service is affected` and providing the service name `Local HPC Resources (e.g. MEG) [CF]` (just type in `Local` and you should get the valid completions).
|
||||
|
||||
### Contact Meg Administrators
|
||||
* PSI HelpDesk will redirect the incident to the corresponding department, or
|
||||
* you can always assign it directly by checking the box `I know which service
|
||||
is affected` and providing the service name `Local HPC Resources (e.g. MEG)
|
||||
[CF]` (just type in `Local` and you should get the valid completions).
|
||||
|
||||
## Contact Meg Administrators
|
||||
|
||||
**E-Mail <meg-admins@lists.psi.ch>** or **<merlin-admins@lists.psi.ch>**
|
||||
|
||||
* This is the preferred way to contact MEG Administrators.
|
||||
Do not hesitate to contact us for such cases.
|
||||
|
||||
---
|
||||
|
||||
## Get updated through the Merlin User list!
|
||||
## Get updated through the Merlin User list
|
||||
|
||||
Is strongly recommended that users subscribe to the Merlin Users mailing list: **<merlin-users@lists.psi.ch>**
|
||||
Is strongly recommended that users subscribe to the Merlin Users mailing list:
|
||||
**<merlin-users@lists.psi.ch>**
|
||||
|
||||
This mailing list is the official channel used by Merlin administrators to inform users about downtimes,
|
||||
interventions or problems. Users can be subscribed in two ways:
|
||||
This mailing list is the official channel used by Merlin administrators to
|
||||
inform users about downtimes, interventions or problems. Users can be
|
||||
subscribed in two ways:
|
||||
|
||||
* *(Preferred way)* Self-registration through **[Sympa](https://psilists.ethz.ch/sympa/info/merlin-users)**
|
||||
* If you need to subscribe many people (e.g. your whole group) by sending a request to the admin list **<merlin-admins@lists.psi.ch>**
|
||||
@@ -46,5 +43,9 @@ interventions or problems. Users can be subscribed in two ways:
|
||||
|
||||
## The MEG Cluster Team
|
||||
|
||||
The PSI Merlin and MEG clusters are managed by the **[High Performance Computing and Emerging technologies Group](https://www.psi.ch/de/lsm/hpce-group)**, which
|
||||
is part of the [Science IT Infrastructure, and Services department (AWI)](https://www.psi.ch/en/awi) in PSI's [Center for Scientific Computing, Theory and Data (SCD)](https://www.psi.ch/en/csd).
|
||||
The PSI Merlin and MEG clusters are managed by the **[High Performance
|
||||
Computing and Emerging technologies
|
||||
Group](https://www.psi.ch/de/lsm/hpce-group)**, which is part of the [Science
|
||||
IT Infrastructure, and Services department (AWI)](https://www.psi.ch/en/awi) in
|
||||
PSI's [Center for Scientific Computing, Theory and Data
|
||||
(SCD)](https://www.psi.ch/en/csd).
|
||||
13
docs/meg/index.md
Normal file
13
docs/meg/index.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# The MEG local HPC cluster
|
||||
|
||||
> The MEG II collaboration includes almost 70 physicists from research
|
||||
> institutions from five countries. Researchers and technicians from PSI have
|
||||
> played a leading role, particularly with providing the high-quality beam,
|
||||
> technical support in the detector integration, and in the design, construction,
|
||||
> and operation of the detector readout electronics."
|
||||
>
|
||||
> —— [Source](https://www.psi.ch/en/cnm/news/in-search-of-new-physics-new-result-from-the-meg-ii-collaboration)
|
||||
|
||||
The MEG data analysis cluster is a cluster tightly coupled to Merlin and
|
||||
dedicated to the analysis of data from the MEG experiment. Operated for the
|
||||
Muon Physics group.
|
||||
@@ -1,12 +1,3 @@
|
||||
---
|
||||
#tags:
|
||||
keywords: meg, merlin6, merlin7, migration, fpsync, rsync
|
||||
#summary: ""
|
||||
sidebar: meg_sidebar
|
||||
last_updated: 28 May 2025
|
||||
permalink: /meg/migrating.html
|
||||
---
|
||||
|
||||
# Meg to Merlin7 Migration Guide
|
||||
|
||||
Welcome to the official documentation for migrating experiment data from **MEG** to **Merlin7**. Please follow the instructions carefully to ensure a smooth and secure transition.
|
||||
@@ -44,20 +35,23 @@ Welcome to the official documentation for migrating experiment data from **MEG**
|
||||
A `experiment_migration.setup` migration script must be executed from **any MeG node** using the account that will perform the migration.
|
||||
|
||||
#### When using the local `root` account
|
||||
- The script **must be executed after every reboot** of the destination nodes.
|
||||
- **Reason:** On Merlin7, the home directory for the `root` user resides on ephemeral storage (no physical disk).
|
||||
|
||||
* The script **must be executed after every reboot** of the destination nodes.
|
||||
* **Reason:** On Merlin7, the home directory for the `root` user resides on ephemeral storage (no physical disk).
|
||||
After a reboot, this directory is cleaned, so **SSH keys need to be redeployed** before running the migration again.
|
||||
|
||||
#### When using a PSI Active Directory (AD) account
|
||||
- Applicable accounts include, for example:
|
||||
- `gac-meg2_data`
|
||||
- `gac-meg2`
|
||||
- The script only needs to be executed **once**, provided that:
|
||||
- The home directory for the AD account is located on a shared storage area.
|
||||
- This shared storage is accessible from the node executing the transfer.
|
||||
- **Reason:** On Merlin7, these accounts have their home directories on persistent shared storage, so the SSH keys remain available across reboots.
|
||||
|
||||
* Applicable accounts include, for example:
|
||||
* `gac-meg2_data`
|
||||
* `gac-meg2`
|
||||
* The script only needs to be executed **once**, provided that:
|
||||
* The home directory for the AD account is located on a shared storage area.
|
||||
* This shared storage is accessible from the node executing the transfer.
|
||||
* **Reason:** On Merlin7, these accounts have their home directories on persistent shared storage, so the SSH keys remain available across reboots.
|
||||
|
||||
To run it:
|
||||
|
||||
```bash
|
||||
experiment_migration.setup
|
||||
```
|
||||
@@ -97,14 +91,15 @@ If you are stuck, email: [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists
|
||||
6. Enable access on the destination folder.
|
||||
* At this point, **no new syncs have to be performed.**
|
||||
|
||||
> ⚠️ **Important Notes**
|
||||
> The `-E` option is destructive; handle with care.
|
||||
> Always verify that the destination is ready before triggering the final sync.
|
||||
> For optimal performance, use up to 12 threads with the -t option.
|
||||
!!! note "Important"
|
||||
|
||||
The `-E` option is destructive; handle with care.
|
||||
Always verify that the destination is ready before triggering the final sync.
|
||||
For optimal performance, use up to 12 threads with the -t option.
|
||||
|
||||
#### Running The Migration Script
|
||||
|
||||
The migration script is installed on the `meg-s-001` server at:
|
||||
The migration script is installed on the `meg-s-001` server at:
|
||||
`/usr/local/bin/experiment_migration.bash`
|
||||
|
||||
This script is primarily a **wrapper** around `fpsync`, providing additional logic for synchronizing MeG experiment data.
|
||||
@@ -143,7 +138,9 @@ Options:
|
||||
-v | --verbose Run fpsync with -v option
|
||||
```
|
||||
|
||||
> Defaults can be updated if necessary.
|
||||
!!! tip
|
||||
|
||||
Defaults can be updated if necessary.
|
||||
|
||||
#### Migration examples
|
||||
|
||||
@@ -151,10 +148,11 @@ Options:
|
||||
|
||||
The following example demonstrates how to migrate the **entire `online`** directory.
|
||||
|
||||
{{site.data.alerts.tip}}
|
||||
You may also choose to migrate only specific subdirectories if needed.
|
||||
However, migrating full directories is generally <b>simpler</b> and <b>less error-prone</b> compared to handling multiple subdirectory migrations.
|
||||
{{site.data.alerts.end}}
|
||||
!!! tip
|
||||
|
||||
You may also choose to migrate only specific subdirectories if needed.
|
||||
However, migrating full directories is generally **simpler** and **less
|
||||
error-prone** compared to handling multiple subdirectory migrations.
|
||||
|
||||
```bash
|
||||
[root@meg-s-001 ~]# experiment_migration.bash -S data1 -D data1 -p "online"
|
||||
@@ -172,9 +170,10 @@ Please confirm to start (y/N):
|
||||
|
||||
The following example demonstrates how to migrate **only a subdirectory**. In this case, we use the option `-F` to create the parent directory in the destination, to ensure that this exists before transferring:
|
||||
|
||||
⚠️ **Important:**
|
||||
- When migrating a subdirectory, **do not** run concurrent migrations on its parent directories.
|
||||
- For example, avoid running migrations with `-p "shared"` while simultaneously migrating `-p "shared/subprojects"`.
|
||||
⚠️ **Important:**
|
||||
|
||||
* When migrating a subdirectory, **do not** run concurrent migrations on its parent directories.
|
||||
* For example, avoid running migrations with `-p "shared"` while simultaneously migrating `-p "shared/subprojects"`.
|
||||
|
||||
```bash
|
||||
[root@meg-s-001 ~]# experiment_migration.bash -p "shared/subprojects/meg1" -F
|
||||
@@ -188,12 +187,14 @@ Please confirm to start (y/N): N
|
||||
❌ Transfer cancelled by user.
|
||||
```
|
||||
|
||||
This command initiates the migration of the directory, by creating the destination parant directory (`-F` option):
|
||||
This command initiates the migration of the directory, by creating the destination parant directory (`-F` option):
|
||||
|
||||
* Creates the destination directory as follows:
|
||||
|
||||
```bash
|
||||
ssh login002.merlin.psi.ch mkdir -p /data/project/meg/data1/shared/subprojects
|
||||
```
|
||||
|
||||
* Runs FPSYNC with 10 threads and N parts of max 20000 files or 100G files:
|
||||
* Source: `/meg/data1/shared/subprojects/meg1`
|
||||
* Source: `/meg/data1/shared/subprojects/meg1`
|
||||
* Destination: `login002.merlin7.psi.ch:/data/project/meg/data1/shared/subprojects/meg1`
|
||||
Reference in New Issue
Block a user