Update Data Transfer to use datatransfer.psi.ch
This commit is contained in:
parent
a0e38e612f
commit
b63ecb4141
@ -21,8 +21,8 @@ visibility.
|
|||||||
- HTTP-based protocols using ports 80 or 445 (https, WebDav, etc)
|
- HTTP-based protocols using ports 80 or 445 (https, WebDav, etc)
|
||||||
- Protocols using other ports require admin configuration and may only work with
|
- Protocols using other ports require admin configuration and may only work with
|
||||||
specific hosts (ftp, rsync daemons, etc)
|
specific hosts (ftp, rsync daemons, etc)
|
||||||
- Systems on the internet can access the Remote Access Merlin servers
|
- Systems on the internet can access the [PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer) service
|
||||||
(ra-merlin\*.psi.ch) using ssh-based protocols
|
`datatransfer.psi.ch`, using ssh-based protocols and [Globus](https://www.globus.org/)
|
||||||
|
|
||||||
|
|
||||||
## Direct transfer via Merlin6 login nodes
|
## Direct transfer via Merlin6 login nodes
|
||||||
@ -68,32 +68,38 @@ The purpose of the software is to send a large file to someone, have that file a
|
|||||||
|
|
||||||
**[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is fully integrated with PSI, therefore, PSI employees can log in by using their PSI account (through Authentication and Authorization Infrastructure / AAI, by selecting PSI as the institution to be used for log in).
|
**[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is fully integrated with PSI, therefore, PSI employees can log in by using their PSI account (through Authentication and Authorization Infrastructure / AAI, by selecting PSI as the institution to be used for log in).
|
||||||
|
|
||||||
## Remote Access Servers
|
## PSI Data Transfer
|
||||||
|
|
||||||
Two servers are enabled for data transfers originating from outside PSI.
|
From August 2024, Merlin is connected to the **[PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)** service,
|
||||||
This is a central service managed by a different team, which is managing the different Remote Access
|
`datatransfer.psi.ch`. This is a central service managed by the Linux team. However, any problems or questions related to it can be directly
|
||||||
services at PSI for the different facilities (including the one for Merlin). However, any problems
|
[reported](/merlin6/contact.html) to the Merlin adminstrators, which will forward the request if necessary.
|
||||||
or questions related to it can be directly [reported](/merlin6/contact.html) to the Merlin adminstrators,
|
|
||||||
which will forward the request if necessary.
|
|
||||||
|
|
||||||
These Remote Access Merlin servers are the following:
|
The PSI Data Transfer servers supports the following protocols:
|
||||||
* **`ra-merlin-01.psi.ch`**
|
* Data Transfer - SSH (scp / rsync)
|
||||||
* **`ra-merlin-02.psi.ch`**
|
* Data Transfer - Globus
|
||||||
|
|
||||||
Both servers have mounted the following Merlin filesystems:
|
Notice that `datatransfer.psi.ch` does not allow SSH login, only `rsync`, `scp` and [Globus](https://www.globus.org/) access is allowed.
|
||||||
* `/data/project` directories mounted in RW on demand. Project responsibles must request it.
|
|
||||||
* `/data/user` mounted in RW (read-write)
|
|
||||||
* `/data/experiment/mu3e` directories mounted in RW (read-write), except `data` (read-only mounted)
|
|
||||||
* `/export` directory in RW (read-write). `/export` is also visible from login nodes.
|
|
||||||
|
|
||||||
Access to the Remote Access server uses ***Multi factor authentication*** (MFA).
|
The following filesystems are mounted:
|
||||||
|
* `/merlin/bio` which points to the `/data/project/bio` directories in Merlin.
|
||||||
|
* `/merlin/user` which points to the `/data/user` directories in Merlin.
|
||||||
|
* `/merlin/export` which points to the `/export` directory in Merlin.
|
||||||
|
* `/merlin/cscs` which points to the `/data/project/general/cscs` directories in Merlin.
|
||||||
|
* `/merlin/mu3e` which points to the `/data/experiment/mu3e` directories in Merlin.
|
||||||
|
* Mu3e sub-directories are mounted in RW (read-write), except for `data` (read-only mounted)
|
||||||
|
|
||||||
|
Access to the PSI Data Transfer uses ***Multi factor authentication*** (MFA).
|
||||||
Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa).
|
Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa).
|
||||||
|
|
||||||
|
{{site.data.alerts.tip}}Please follow the
|
||||||
|
<b><a href="https://www.psi.ch/en/photon-science-data-services/data-transfer">Official PSI Data Transfer</a></b> documentation for further instructions.
|
||||||
|
{{site.data.alerts.end}}
|
||||||
|
|
||||||
### Directories
|
### Directories
|
||||||
|
|
||||||
#### /data/user
|
#### /merlin/user
|
||||||
|
|
||||||
User data directories are mounted in RW on both 'ra-merlin-01' and 'ra-merlin-02'.
|
User data directories are mounted in RW.
|
||||||
|
|
||||||
{{site.data.alerts.warning}}Please, <b>ensure proper secured permissions</b> in your '/data/user'
|
{{site.data.alerts.warning}}Please, <b>ensure proper secured permissions</b> in your '/data/user'
|
||||||
directory. By default, when directory is created, the system applies the most restrictive
|
directory. By default, when directory is created, the system applies the most restrictive
|
||||||
@ -101,7 +107,7 @@ permissions. However, this does not prevent users for changing permissions if th
|
|||||||
point, users become responsible of those changes.
|
point, users become responsible of those changes.
|
||||||
{{site.data.alerts.end}}
|
{{site.data.alerts.end}}
|
||||||
|
|
||||||
#### /export
|
#### /merlin/export
|
||||||
|
|
||||||
Transferring big amounts of data from outside PSI to Merlin is always possible through `/export`.
|
Transferring big amounts of data from outside PSI to Merlin is always possible through `/export`.
|
||||||
|
|
||||||
@ -118,41 +124,23 @@ This is configured in Read/Write mode. If you need access, please, contact the M
|
|||||||
For exporting data from Merlin to outside PSI by using `/export`, one has to:
|
For exporting data from Merlin to outside PSI by using `/export`, one has to:
|
||||||
* From a Merlin login node, copy your data from any directory (i.e. `/data/project`, `/data/user`, `/scratch`) to
|
* From a Merlin login node, copy your data from any directory (i.e. `/data/project`, `/data/user`, `/scratch`) to
|
||||||
`/export`. Ensure to properly secure your directories and files with proper permissions.
|
`/export`. Ensure to properly secure your directories and files with proper permissions.
|
||||||
* Once data is copied, from **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from `/export` to outside PSI.
|
* Once data is copied, from **`datatransfer.psi.ch`**, copy the data from `/merlin/export` to outside PSI
|
||||||
|
|
||||||
##### Importing data to Merlin
|
##### Importing data to Merlin
|
||||||
|
|
||||||
For importing data from outside PSI to Merlin by using `/export`, one has to:
|
For importing data from outside PSI to Merlin by using `/export`, one has to:
|
||||||
* From **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from outside PSI to `/export`.
|
* From **`datatransfer.psi.ch`**, copy the data from outside PSI to `/merlin/export`.
|
||||||
Ensure to properly secure your directories and files with proper permissions.
|
Ensure to properly secure your directories and files with proper permissions.
|
||||||
* Once data is copied, from a Merlin login node, copy your data from `/export` to any directory (i.e. `/data/project`, `/data/user`, `/scratch`).
|
* Once data is copied, from a Merlin login node, copy your data from `/export` to any directory (i.e. `/data/project`, `/data/user`, `/scratch`).
|
||||||
|
|
||||||
#### /data/project
|
#### Request access to your project directory
|
||||||
|
|
||||||
Optionally, instead of using `/export`, experiments with a Merlin project can request Read/Write or Read/Only access to their project directory.
|
Optionally, instead of using `/export`, Merlin project owners can request Read/Write or Read/Only access to their project directory.
|
||||||
|
|
||||||
{{site.data.alerts.tip}}<b>Merlin projects can request direct access.</b>
|
{{site.data.alerts.tip}}<b>Merlin projects can request direct access.</b>
|
||||||
This can be configured in Read/Write or Read/Only modes. If your project needs access, please,
|
This can be configured in Read/Write or Read/Only modes. If your project needs access, please, contact the Merlin administrators.
|
||||||
contact the Merlin administrators.
|
|
||||||
{{site.data.alerts.end}}
|
{{site.data.alerts.end}}
|
||||||
|
|
||||||
### Accepted protocols
|
|
||||||
|
|
||||||
Accepted protocols for Remote Access Merlin servers are the following:
|
|
||||||
* **sftp**: **``sftp``** command or similar X11/Windows/MacOS based programs.
|
|
||||||
* **ssh**: **`scp`** command (as well as **WinSCP** and similar programs) or **`rsync`** command
|
|
||||||
* **~~Globus Online~~**: ***not available yet.***
|
|
||||||
|
|
||||||
### Remote Access Servers Policies
|
|
||||||
|
|
||||||
SSH is one of the allowed protocols.
|
|
||||||
* Please, **absolutely never** use this servers as a login node.
|
|
||||||
* Please avoid copying files to the *home* directories.
|
|
||||||
* Please **never use SSH Keys** for accessing these servers. Accessing through SSH keys will be denied in the upcomig months.
|
|
||||||
|
|
||||||
Only ``/data/user`, `/data/project` and `/export` directories should be used on these nodes,
|
|
||||||
and exclusively for transferring data from/to PSI to/from outside PSI.
|
|
||||||
|
|
||||||
## Connecting to Merlin6 from outside PSI
|
## Connecting to Merlin6 from outside PSI
|
||||||
|
|
||||||
Merlin6 is fully accessible from within the PSI network. To connect from outside you can use:
|
Merlin6 is fully accessible from within the PSI network. To connect from outside you can use:
|
||||||
|
Loading…
x
Reference in New Issue
Block a user