HPCE/gitea-pages
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Activity

Update Data Transfer to use datatransfer.psi.ch

Browse Source
This commit is contained in:
caubet_m
2024-07-31 13:52:28 +02:00
parent a0e38e612f
commit b63ecb4141
1 changed files with 31 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
pages/merlin6/02-How-To-Use-Merlin/transfer-data.md
View File

@ -21,8 +21,8 @@ visibility.
- HTTP-based protocols using ports 80 or 445 (https, WebDav, etc) - HTTP-based protocols using ports 80 or 445 (https, WebDav, etc)
- Protocols using other ports require admin configuration and may only work with - Protocols using other ports require admin configuration and may only work with
specific hosts (ftp, rsync daemons, etc) specific hosts (ftp, rsync daemons, etc)
- Systems on the internet can access the Remote Access Merlin servers - Systems on the internet can access the [PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer) service
(ra-merlin\*.psi.ch) using ssh-based protocols `datatransfer.psi.ch`, using ssh-based protocols and [Globus](https://www.globus.org/)
## Direct transfer via Merlin6 login nodes ## Direct transfer via Merlin6 login nodes
@ -68,32 +68,38 @@ The purpose of the software is to send a large file to someone, have that file a
**[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is fully integrated with PSI, therefore, PSI employees can log in by using their PSI account (through Authentication and Authorization Infrastructure / AAI, by selecting PSI as the institution to be used for log in). **[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is fully integrated with PSI, therefore, PSI employees can log in by using their PSI account (through Authentication and Authorization Infrastructure / AAI, by selecting PSI as the institution to be used for log in).
## Remote Access Servers ## PSI Data Transfer
Two servers are enabled for data transfers originating from outside PSI. From August 2024, Merlin is connected to the **[PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)** service,
This is a central service managed by a different team, which is managing the different Remote Access `datatransfer.psi.ch`. This is a central service managed by the Linux team. However, any problems or questions related to it can be directly
services at PSI for the different facilities (including the one for Merlin). However, any problems [reported](/merlin6/contact.html) to the Merlin adminstrators, which will forward the request if necessary.
or questions related to it can be directly [reported](/merlin6/contact.html) to the Merlin adminstrators,
which will forward the request if necessary.
These Remote Access Merlin servers are the following: The PSI Data Transfer servers supports the following protocols:
* **`ra-merlin-01.psi.ch`** * Data Transfer - SSH (scp / rsync)
* **`ra-merlin-02.psi.ch`** * Data Transfer - Globus
Both servers have mounted the following Merlin filesystems: Notice that `datatransfer.psi.ch` does not allow SSH login, only `rsync`, `scp` and [Globus](https://www.globus.org/) access is allowed.
* `/data/project` directories mounted in RW on demand. Project responsibles must request it.
* `/data/user` mounted in RW (read-write)
* `/data/experiment/mu3e` directories mounted in RW (read-write), except `data` (read-only mounted)
* `/export` directory in RW (read-write). `/export` is also visible from login nodes.
Access to the Remote Access server uses ***Multi factor authentication*** (MFA). The following filesystems are mounted:
* `/merlin/bio` which points to the `/data/project/bio` directories in Merlin.
* `/merlin/user` which points to the `/data/user` directories in Merlin.
* `/merlin/export` which points to the `/export` directory in Merlin.
* `/merlin/cscs` which points to the `/data/project/general/cscs` directories in Merlin.
* `/merlin/mu3e` which points to the `/data/experiment/mu3e` directories in Merlin.
* Mu3e sub-directories are mounted in RW (read-write), except for `data` (read-only mounted)
Access to the PSI Data Transfer uses ***Multi factor authentication*** (MFA).
Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa). Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa).
{{site.data.alerts.tip}}Please follow the
<b><a href="https://www.psi.ch/en/photon-science-data-services/data-transfer">Official PSI Data Transfer</a></b> documentation for further instructions.
{{site.data.alerts.end}}
### Directories ### Directories
#### /data/user #### /merlin/user
User data directories are mounted in RW on both 'ra-merlin-01' and 'ra-merlin-02'. User data directories are mounted in RW.
{{site.data.alerts.warning}}Please, <b>ensure proper secured permissions</b> in your '/data/user' {{site.data.alerts.warning}}Please, <b>ensure proper secured permissions</b> in your '/data/user'
directory. By default, when directory is created, the system applies the most restrictive directory. By default, when directory is created, the system applies the most restrictive
@ -101,7 +107,7 @@ permissions. However, this does not prevent users for changing permissions if th
point, users become responsible of those changes. point, users become responsible of those changes.
{{site.data.alerts.end}} {{site.data.alerts.end}}
#### /export #### /merlin/export
Transferring big amounts of data from outside PSI to Merlin is always possible through `/export`. Transferring big amounts of data from outside PSI to Merlin is always possible through `/export`.
@ -118,41 +124,23 @@ This is configured in Read/Write mode. If you need access, please, contact the M
For exporting data from Merlin to outside PSI by using `/export`, one has to: For exporting data from Merlin to outside PSI by using `/export`, one has to:
* From a Merlin login node, copy your data from any directory (i.e. `/data/project`, `/data/user`, `/scratch`) to * From a Merlin login node, copy your data from any directory (i.e. `/data/project`, `/data/user`, `/scratch`) to
`/export`. Ensure to properly secure your directories and files with proper permissions. `/export`. Ensure to properly secure your directories and files with proper permissions.
* Once data is copied, from **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from `/export` to outside PSI. * Once data is copied, from **`datatransfer.psi.ch`**, copy the data from `/merlin/export` to outside PSI
##### Importing data to Merlin ##### Importing data to Merlin
For importing data from outside PSI to Merlin by using `/export`, one has to: For importing data from outside PSI to Merlin by using `/export`, one has to:
* From **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from outside PSI to `/export`. * From **`datatransfer.psi.ch`**, copy the data from outside PSI to `/merlin/export`.
Ensure to properly secure your directories and files with proper permissions. Ensure to properly secure your directories and files with proper permissions.
* Once data is copied, from a Merlin login node, copy your data from `/export` to any directory (i.e. `/data/project`, `/data/user`, `/scratch`). * Once data is copied, from a Merlin login node, copy your data from `/export` to any directory (i.e. `/data/project`, `/data/user`, `/scratch`).
#### /data/project #### Request access to your project directory
Optionally, instead of using `/export`, experiments with a Merlin project can request Read/Write or Read/Only access to their project directory. Optionally, instead of using `/export`, Merlin project owners can request Read/Write or Read/Only access to their project directory.
{{site.data.alerts.tip}}<b>Merlin projects can request direct access.</b> {{site.data.alerts.tip}}<b>Merlin projects can request direct access.</b>
This can be configured in Read/Write or Read/Only modes. If your project needs access, please, This can be configured in Read/Write or Read/Only modes. If your project needs access, please, contact the Merlin administrators.
contact the Merlin administrators.
{{site.data.alerts.end}} {{site.data.alerts.end}}
### Accepted protocols
Accepted protocols for Remote Access Merlin servers are the following:
* **sftp**: **``sftp``** command or similar X11/Windows/MacOS based programs.
* **ssh**: **`scp`** command (as well as **WinSCP** and similar programs) or **`rsync`** command
* **~~Globus Online~~**: ***not available yet.***
### Remote Access Servers Policies
SSH is one of the allowed protocols.
* Please, **absolutely never** use this servers as a login node.
* Please avoid copying files to the *home* directories.
* Please **never use SSH Keys** for accessing these servers. Accessing through SSH keys will be denied in the upcomig months.
Only ``/data/user`, `/data/project` and `/export` directories should be used on these nodes,
and exclusively for transferring data from/to PSI to/from outside PSI.
## Connecting to Merlin6 from outside PSI ## Connecting to Merlin6 from outside PSI
Merlin6 is fully accessible from within the PSI network. To connect from outside you can use: Merlin6 is fully accessible from within the PSI network. To connect from outside you can use: