refactor CSCS and Meg content
This commit is contained in:
@@ -1,15 +0,0 @@
|
||||
tip: '<div class="alert alert-success" role="alert"><i class="fa fa-check-square-o"></i> <b>Tip: </b>'
|
||||
note: '<div class="alert alert-info" role="alert"><i class="fa fa-info-circle"></i> <b>Note: </b>'
|
||||
important: '<div class="alert alert-warning" role="alert"><i class="fa fa-warning"></i> <b>Important: </b>'
|
||||
warning: '<div class="alert alert-danger" role="alert"><i class="fa fa-exclamation-circle"></i> <b>Warning: </b>'
|
||||
end: '</div>'
|
||||
|
||||
callout_danger: '<div class="bs-callout bs-callout-danger">'
|
||||
callout_default: '<div class="bs-callout bs-callout-default">'
|
||||
callout_primary: '<div class="bs-callout bs-callout-primary">'
|
||||
callout_success: '<div class="bs-callout bs-callout-success">'
|
||||
callout_info: '<div class="bs-callout bs-callout-info">'
|
||||
callout_warning: '<div class="bs-callout bs-callout-warning">'
|
||||
|
||||
hr_faded: '<hr class="faded"/>'
|
||||
hr_shaded: '<hr class="shaded"/>'
|
||||
@@ -1,9 +0,0 @@
|
||||
elephant: "This is a sample definition."
|
||||
|
||||
baseball: "Baseball is considered America's pasttime sport, though that may be more of a historical term than a current one. There's a lot more excitement about football than baseball. A baseball game is somewhat of a snooze to watch, for the most part."
|
||||
|
||||
basketball: "Basketball is a sport involving two teams of five players each competing to put a ball through a small circular rim 10 feet above the ground. Basketball requires players to be in top physical condition, since they spend most of the game running back and forth along a 94-foot-long floor."
|
||||
|
||||
football: "No doubt the most fun sport to watch, football also manages to accrue the most injuries with the players. From concussions to blown knees, football players have short sport lives."
|
||||
|
||||
soccer: "If there's one sport that dominates the world landscape, it's soccer. However, US soccer fans are few and far between. Apart from the popularity of soccer during the World Cup, most people don't even know the name of the professional soccer organization in their area."
|
||||
@@ -1,11 +0,0 @@
|
||||
jekyll_platform: "Jekyll is a static site generator that builds sites using most modern web technologies."
|
||||
|
||||
fractious: "Like a little mischevious child, full of annoying and constant trouble."
|
||||
|
||||
gratuitous: "Something that is unwarranted and uncouth, like the social equivalent of a flagrant foul."
|
||||
|
||||
haughty: "Proud and flaunting it. Holding your head high up like a snooty, too-good-for-everything rich person."
|
||||
|
||||
impertinent: "Someone acting rude and insensitive to others."
|
||||
|
||||
intrepid: "Brave and courageous especially in a difficult, dangerous situation."
|
||||
@@ -1,107 +0,0 @@
|
||||
entries:
|
||||
- title: Sidebar
|
||||
folders:
|
||||
- title: Food
|
||||
|
||||
folderitems:
|
||||
- title: Bananas
|
||||
url: bananas.html
|
||||
|
||||
subfolders:
|
||||
- title: Apples
|
||||
|
||||
subfolderitems:
|
||||
- title: Fuji apples
|
||||
url: fuji_apples.html
|
||||
|
||||
|
||||
- title: Gala apples
|
||||
url: gala_apples.html
|
||||
|
||||
name:
|
||||
husband: Tom
|
||||
wife: Shannon
|
||||
|
||||
bikes:
|
||||
- title: mountain bikes
|
||||
- title: road bikes
|
||||
- title: hybrid bikes
|
||||
|
||||
|
||||
salesteams:
|
||||
- title: Regions
|
||||
subfolderitems:
|
||||
- location: US
|
||||
- location: Spain
|
||||
- location: France
|
||||
|
||||
toc:
|
||||
- title: Group 1
|
||||
subfolderitems:
|
||||
- page: Thing 1
|
||||
- page: Thing 2
|
||||
- page: Thing 3
|
||||
- title: Group 2
|
||||
subfolderitems:
|
||||
- page: Piece 1
|
||||
- page: Piece 2
|
||||
- page: Piece 3
|
||||
- title: Group 3
|
||||
subfolderitems:
|
||||
- page: Widget 1
|
||||
- page: Widget 2
|
||||
- page: Widget 3
|
||||
|
||||
something: &hello Greetings earthling!
|
||||
myref: *hello
|
||||
|
||||
about:
|
||||
- zero
|
||||
- one
|
||||
- two
|
||||
- three
|
||||
|
||||
numbercolors:
|
||||
- zero:
|
||||
properties: red
|
||||
- one:
|
||||
properties: yellow
|
||||
- two:
|
||||
properties: green
|
||||
- three:
|
||||
properties: blue
|
||||
|
||||
mypages:
|
||||
- section1: Section 1
|
||||
audience: developers
|
||||
product: acme
|
||||
url: facebook.com
|
||||
- section2: Section 2
|
||||
audience: writers
|
||||
product: acme
|
||||
url: google.com
|
||||
- section3: Section 3
|
||||
audience: developers
|
||||
product: acme
|
||||
url: amazon.com
|
||||
- section4: Section 4
|
||||
audience: writers
|
||||
product: gizmo
|
||||
url: apple.com
|
||||
- section5: Section 5
|
||||
audience: writers
|
||||
product: acme
|
||||
url: microsoft.com
|
||||
|
||||
feedback: >
|
||||
This is my feedback to you.
|
||||
Even if I include linebreaks here,
|
||||
all of the linebreaks will be removed when the value is inserted.
|
||||
|
||||
block: |
|
||||
This pipe does something a little different.
|
||||
It preserves the breaks.
|
||||
This is really helpful for code samples,
|
||||
since you can format the code samples with
|
||||
the appropriate
|
||||
white spacing.
|
||||
@@ -1,15 +0,0 @@
|
||||
|
||||
# Follow the pattern here for the URLs -- no slash at the beginning, and include the .html. The link here is rendered exactly as is in the Markdown references.
|
||||
|
||||
entries:
|
||||
- product: PSI HPC@CSCS
|
||||
folders:
|
||||
- title: Overview
|
||||
# URLs for top-level folders are optional. If omitted it is a bit easier to toggle the accordion.
|
||||
folderitems:
|
||||
- title: Overview
|
||||
url: /CSCS/index.html
|
||||
- title: Operations
|
||||
folderitems:
|
||||
- title: Transfer Data
|
||||
url: /CSCS/transfer-data.html
|
||||
@@ -1,17 +0,0 @@
|
||||
# Follow the pattern here for the URLs -- no slash at the beginning, and include the .html. The link here is rendered exactly as is in the Markdown references.
|
||||
|
||||
entries:
|
||||
- product: MEG
|
||||
version:
|
||||
folders:
|
||||
- title: Quick Start Guide
|
||||
folderitems:
|
||||
- title: Introduction
|
||||
url: /meg/introduction.html
|
||||
- title: Support
|
||||
folderitems:
|
||||
- title: MeG Migration Guide
|
||||
url: /meg/migrating.html
|
||||
- title: Contact
|
||||
url: /meg/contact.html
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
|
||||
|
||||
# placed here for translation purposes
|
||||
search_placeholder_text: search...
|
||||
search_no_results_text: No results found.
|
||||
@@ -1,15 +0,0 @@
|
||||
# Note:
|
||||
# If you are using the createtag script, don't leave an blank line at the end of this file.
|
||||
# In other words, the last line must be the last tag in the allowed-tags list.
|
||||
allowed-tags:
|
||||
- getting_started
|
||||
- content_types
|
||||
- navigation
|
||||
- formatting
|
||||
- publishing
|
||||
- single_sourcing
|
||||
- special_layouts
|
||||
- collaboration
|
||||
- news
|
||||
- troubleshooting
|
||||
- mobile
|
||||
@@ -1,3 +0,0 @@
|
||||
hpc: "High Performance Computing"
|
||||
hpce: "High Performance Computing and Emerging Technologies; our group at PSI"
|
||||
psi: "Paul Scherrer Institute"
|
||||
@@ -1,36 +0,0 @@
|
||||
## Topnav single links
|
||||
## if you want to list an external url, use external_url instead of url. the theme will apply a different link base.
|
||||
topnav:
|
||||
- title: Topnav
|
||||
items:
|
||||
# - title: GitHub
|
||||
# external_url: https://github.com/tomjoht/documentation-theme-jekyll
|
||||
- title: News
|
||||
url: /news.html
|
||||
|
||||
#Topnav dropdowns
|
||||
topnav_dropdowns:
|
||||
- title: Topnav dropdowns
|
||||
folders:
|
||||
- title: Quick Start
|
||||
folderitems:
|
||||
- title: Introduction
|
||||
url: /merlin6/introduction.html
|
||||
- title: Requesting Merlin Access
|
||||
url: /merlin6/request-account.html
|
||||
- title: Requesting Merlin Projects
|
||||
url: /merlin6/request-project.html
|
||||
- title: Accessing the Interactive Nodes
|
||||
url: /merlin6/interactive.html
|
||||
- title: Accessing the Slurm Clusters
|
||||
url: /merlin6/slurm-access.html
|
||||
- title: Clusters
|
||||
folderitems:
|
||||
- title: Merlin 5
|
||||
url: /merlin5/slurm-configuration.html
|
||||
- title: Merlin 6
|
||||
url: /merlin6/slurm-configuration.html
|
||||
- title: Merlin 6 GPU
|
||||
url: /gmerlin6/slurm-configuration.html
|
||||
- title: Merlin 7
|
||||
url: /merlin7/slurm-configuration.html
|
||||
@@ -1,14 +1,4 @@
|
||||
---
|
||||
title: Transferring Data betweem PSI and CSCS
|
||||
#tags:
|
||||
keywords: CSCS, data-transfer
|
||||
last_updated: 02 March 2022
|
||||
summary: "This Document shows the procedure for transferring data between CSCS and PSI"
|
||||
sidebar: CSCS_sidebar
|
||||
permalink: /CSCS/transfer-data.html
|
||||
---
|
||||
|
||||
# Transferring Data
|
||||
# Transferring Data
|
||||
|
||||
This document shows how to transfer data between PSI and CSCS by using a Linux workstation.
|
||||
|
||||
@@ -37,7 +27,6 @@ Host *.cscs.ch
|
||||
There are many different SSH settings available which would allow advanced configurations.
|
||||
Users may have some configurations already present, therefore would need to adapt it accordingly.
|
||||
|
||||
|
||||
## Transferring files
|
||||
|
||||
Once the above configuration is set, then try to rsync from Merlin to CSCS, on any direction:
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
---
|
||||
title: Introduction
|
||||
#tags:
|
||||
keywords: introduction, home, welcome, architecture, design
|
||||
last_updated: 07 September 2022
|
||||
#summary: "MeG cluster overview"
|
||||
sidebar: meg_sidebar
|
||||
permalink: /meg/introduction.html
|
||||
redirect_from:
|
||||
- /meg
|
||||
- /meg/index.html
|
||||
---
|
||||
|
||||
## The Meg local HPC cluster
|
||||
|
||||
"The MEG II collaboration includes almost 70 physicists from research institutions from five countries. Researchers and technicians from PSI have played a leading role, particularly with providing the high-quality beam, technical support in the detector integration, and in the design, construction, and operation of the detector readout electronics." [Source](https://www.psi.ch/en/cnm/news/in-search-of-new-physics-new-result-from-the-meg-ii-collaboration)
|
||||
|
||||
The MEG data analysis cluster is a cluster tightly coupled to Merlin and dedicated to the analysis of data from the MEG experiment. Operated for the Muon Physics group.
|
||||
@@ -1,42 +1,39 @@
|
||||
---
|
||||
title: Contact
|
||||
#tags:
|
||||
keywords: contact, support, snow, service now, mailing list, mailing, email, mail, meg-admins@lists.psi.ch, merlin users
|
||||
last_updated: 15. Jan 2025
|
||||
#summary: ""
|
||||
sidebar: meg_sidebar
|
||||
permalink: /meg/contact.html
|
||||
---
|
||||
|
||||
## Support
|
||||
# Support
|
||||
|
||||
Support can be asked through:
|
||||
|
||||
* [PSI Service Now](https://psi.service-now.com/psisp)
|
||||
* E-Mail: <meg-admins@lists.psi.ch>
|
||||
|
||||
Basic contact information is also displayed on every shell login to the system using the *Message of the Day* mechanism.
|
||||
Basic contact information is also displayed on every shell login to the system
|
||||
using the *Message of the Day* mechanism.
|
||||
|
||||
|
||||
### PSI Service Now
|
||||
## PSI Service Now
|
||||
|
||||
**[PSI Service Now](https://psi.service-now.com/psisp)**: is the official PSI tool for opening incident requests. However, contact via email (see below) is preferred.
|
||||
* PSI HelpDesk will redirect the incident to the corresponding department, or
|
||||
* you can always assign it directly by checking the box `I know which service is affected` and providing the service name `Local HPC Resources (e.g. MEG) [CF]` (just type in `Local` and you should get the valid completions).
|
||||
|
||||
### Contact Meg Administrators
|
||||
* PSI HelpDesk will redirect the incident to the corresponding department, or
|
||||
* you can always assign it directly by checking the box `I know which service
|
||||
is affected` and providing the service name `Local HPC Resources (e.g. MEG)
|
||||
[CF]` (just type in `Local` and you should get the valid completions).
|
||||
|
||||
## Contact Meg Administrators
|
||||
|
||||
**E-Mail <meg-admins@lists.psi.ch>** or **<merlin-admins@lists.psi.ch>**
|
||||
|
||||
* This is the preferred way to contact MEG Administrators.
|
||||
Do not hesitate to contact us for such cases.
|
||||
|
||||
---
|
||||
|
||||
## Get updated through the Merlin User list!
|
||||
## Get updated through the Merlin User list
|
||||
|
||||
Is strongly recommended that users subscribe to the Merlin Users mailing list: **<merlin-users@lists.psi.ch>**
|
||||
Is strongly recommended that users subscribe to the Merlin Users mailing list:
|
||||
**<merlin-users@lists.psi.ch>**
|
||||
|
||||
This mailing list is the official channel used by Merlin administrators to inform users about downtimes,
|
||||
interventions or problems. Users can be subscribed in two ways:
|
||||
This mailing list is the official channel used by Merlin administrators to
|
||||
inform users about downtimes, interventions or problems. Users can be
|
||||
subscribed in two ways:
|
||||
|
||||
* *(Preferred way)* Self-registration through **[Sympa](https://psilists.ethz.ch/sympa/info/merlin-users)**
|
||||
* If you need to subscribe many people (e.g. your whole group) by sending a request to the admin list **<merlin-admins@lists.psi.ch>**
|
||||
@@ -46,5 +43,9 @@ interventions or problems. Users can be subscribed in two ways:
|
||||
|
||||
## The MEG Cluster Team
|
||||
|
||||
The PSI Merlin and MEG clusters are managed by the **[High Performance Computing and Emerging technologies Group](https://www.psi.ch/de/lsm/hpce-group)**, which
|
||||
is part of the [Science IT Infrastructure, and Services department (AWI)](https://www.psi.ch/en/awi) in PSI's [Center for Scientific Computing, Theory and Data (SCD)](https://www.psi.ch/en/csd).
|
||||
The PSI Merlin and MEG clusters are managed by the **[High Performance
|
||||
Computing and Emerging technologies
|
||||
Group](https://www.psi.ch/de/lsm/hpce-group)**, which is part of the [Science
|
||||
IT Infrastructure, and Services department (AWI)](https://www.psi.ch/en/awi) in
|
||||
PSI's [Center for Scientific Computing, Theory and Data
|
||||
(SCD)](https://www.psi.ch/en/csd).
|
||||
13
docs/meg/index.md
Normal file
13
docs/meg/index.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# The MEG local HPC cluster
|
||||
|
||||
> The MEG II collaboration includes almost 70 physicists from research
|
||||
> institutions from five countries. Researchers and technicians from PSI have
|
||||
> played a leading role, particularly with providing the high-quality beam,
|
||||
> technical support in the detector integration, and in the design, construction,
|
||||
> and operation of the detector readout electronics."
|
||||
>
|
||||
> —— [Source](https://www.psi.ch/en/cnm/news/in-search-of-new-physics-new-result-from-the-meg-ii-collaboration)
|
||||
|
||||
The MEG data analysis cluster is a cluster tightly coupled to Merlin and
|
||||
dedicated to the analysis of data from the MEG experiment. Operated for the
|
||||
Muon Physics group.
|
||||
@@ -1,12 +1,3 @@
|
||||
---
|
||||
#tags:
|
||||
keywords: meg, merlin6, merlin7, migration, fpsync, rsync
|
||||
#summary: ""
|
||||
sidebar: meg_sidebar
|
||||
last_updated: 28 May 2025
|
||||
permalink: /meg/migrating.html
|
||||
---
|
||||
|
||||
# Meg to Merlin7 Migration Guide
|
||||
|
||||
Welcome to the official documentation for migrating experiment data from **MEG** to **Merlin7**. Please follow the instructions carefully to ensure a smooth and secure transition.
|
||||
@@ -44,20 +35,23 @@ Welcome to the official documentation for migrating experiment data from **MEG**
|
||||
A `experiment_migration.setup` migration script must be executed from **any MeG node** using the account that will perform the migration.
|
||||
|
||||
#### When using the local `root` account
|
||||
- The script **must be executed after every reboot** of the destination nodes.
|
||||
- **Reason:** On Merlin7, the home directory for the `root` user resides on ephemeral storage (no physical disk).
|
||||
|
||||
* The script **must be executed after every reboot** of the destination nodes.
|
||||
* **Reason:** On Merlin7, the home directory for the `root` user resides on ephemeral storage (no physical disk).
|
||||
After a reboot, this directory is cleaned, so **SSH keys need to be redeployed** before running the migration again.
|
||||
|
||||
#### When using a PSI Active Directory (AD) account
|
||||
- Applicable accounts include, for example:
|
||||
- `gac-meg2_data`
|
||||
- `gac-meg2`
|
||||
- The script only needs to be executed **once**, provided that:
|
||||
- The home directory for the AD account is located on a shared storage area.
|
||||
- This shared storage is accessible from the node executing the transfer.
|
||||
- **Reason:** On Merlin7, these accounts have their home directories on persistent shared storage, so the SSH keys remain available across reboots.
|
||||
|
||||
* Applicable accounts include, for example:
|
||||
* `gac-meg2_data`
|
||||
* `gac-meg2`
|
||||
* The script only needs to be executed **once**, provided that:
|
||||
* The home directory for the AD account is located on a shared storage area.
|
||||
* This shared storage is accessible from the node executing the transfer.
|
||||
* **Reason:** On Merlin7, these accounts have their home directories on persistent shared storage, so the SSH keys remain available across reboots.
|
||||
|
||||
To run it:
|
||||
|
||||
```bash
|
||||
experiment_migration.setup
|
||||
```
|
||||
@@ -97,14 +91,15 @@ If you are stuck, email: [merlin-admins@lists.psi.ch](mailto:merlin-admins@lists
|
||||
6. Enable access on the destination folder.
|
||||
* At this point, **no new syncs have to be performed.**
|
||||
|
||||
> ⚠️ **Important Notes**
|
||||
> The `-E` option is destructive; handle with care.
|
||||
> Always verify that the destination is ready before triggering the final sync.
|
||||
> For optimal performance, use up to 12 threads with the -t option.
|
||||
!!! note "Important"
|
||||
|
||||
The `-E` option is destructive; handle with care.
|
||||
Always verify that the destination is ready before triggering the final sync.
|
||||
For optimal performance, use up to 12 threads with the -t option.
|
||||
|
||||
#### Running The Migration Script
|
||||
|
||||
The migration script is installed on the `meg-s-001` server at:
|
||||
The migration script is installed on the `meg-s-001` server at:
|
||||
`/usr/local/bin/experiment_migration.bash`
|
||||
|
||||
This script is primarily a **wrapper** around `fpsync`, providing additional logic for synchronizing MeG experiment data.
|
||||
@@ -143,7 +138,9 @@ Options:
|
||||
-v | --verbose Run fpsync with -v option
|
||||
```
|
||||
|
||||
> Defaults can be updated if necessary.
|
||||
!!! tip
|
||||
|
||||
Defaults can be updated if necessary.
|
||||
|
||||
#### Migration examples
|
||||
|
||||
@@ -151,10 +148,11 @@ Options:
|
||||
|
||||
The following example demonstrates how to migrate the **entire `online`** directory.
|
||||
|
||||
{{site.data.alerts.tip}}
|
||||
You may also choose to migrate only specific subdirectories if needed.
|
||||
However, migrating full directories is generally <b>simpler</b> and <b>less error-prone</b> compared to handling multiple subdirectory migrations.
|
||||
{{site.data.alerts.end}}
|
||||
!!! tip
|
||||
|
||||
You may also choose to migrate only specific subdirectories if needed.
|
||||
However, migrating full directories is generally **simpler** and **less
|
||||
error-prone** compared to handling multiple subdirectory migrations.
|
||||
|
||||
```bash
|
||||
[root@meg-s-001 ~]# experiment_migration.bash -S data1 -D data1 -p "online"
|
||||
@@ -172,9 +170,10 @@ Please confirm to start (y/N):
|
||||
|
||||
The following example demonstrates how to migrate **only a subdirectory**. In this case, we use the option `-F` to create the parent directory in the destination, to ensure that this exists before transferring:
|
||||
|
||||
⚠️ **Important:**
|
||||
- When migrating a subdirectory, **do not** run concurrent migrations on its parent directories.
|
||||
- For example, avoid running migrations with `-p "shared"` while simultaneously migrating `-p "shared/subprojects"`.
|
||||
⚠️ **Important:**
|
||||
|
||||
* When migrating a subdirectory, **do not** run concurrent migrations on its parent directories.
|
||||
* For example, avoid running migrations with `-p "shared"` while simultaneously migrating `-p "shared/subprojects"`.
|
||||
|
||||
```bash
|
||||
[root@meg-s-001 ~]# experiment_migration.bash -p "shared/subprojects/meg1" -F
|
||||
@@ -188,12 +187,14 @@ Please confirm to start (y/N): N
|
||||
❌ Transfer cancelled by user.
|
||||
```
|
||||
|
||||
This command initiates the migration of the directory, by creating the destination parant directory (`-F` option):
|
||||
This command initiates the migration of the directory, by creating the destination parant directory (`-F` option):
|
||||
|
||||
* Creates the destination directory as follows:
|
||||
|
||||
```bash
|
||||
ssh login002.merlin.psi.ch mkdir -p /data/project/meg/data1/shared/subprojects
|
||||
```
|
||||
|
||||
* Runs FPSYNC with 10 threads and N parts of max 20000 files or 100G files:
|
||||
* Source: `/meg/data1/shared/subprojects/meg1`
|
||||
* Source: `/meg/data1/shared/subprojects/meg1`
|
||||
* Destination: `login002.merlin7.psi.ch:/data/project/meg/data1/shared/subprojects/meg1`
|
||||
@@ -80,5 +80,8 @@ nav:
|
||||
- merlin6/index.md
|
||||
- PSI@CSCS:
|
||||
- cscs-userlab/index.md
|
||||
- cscs-userlab/transfer-data.md
|
||||
- MeG:
|
||||
- meg/index.md
|
||||
- meg/contact.md
|
||||
- meg/migration-to-merlin7.md
|
||||
|
||||
Reference in New Issue
Block a user