HPCE/gitea-pages
0
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Activity

Improve docs
Build and deploy documentation / build-and-deploy-docs (push) Successful in 20s
Details

Browse Source
This commit is contained in:
caubet_m
2026-03-31 15:58:53 +02:00
parent 93cfd9b958
commit b0d3dd2f8c
1 changed files with 76 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/merlin7/02-How-To-Use-Merlin/transfer-data.md
+76 -66
View File
@@ -32,11 +32,23 @@ Most data transfer methods support both sending and receiving, so you may initia
The next chapters contain detailed information about the different transfer methods available on Merlin7.
## Direct Transfer via Merlin7 Login Nodes
## Transferring from within PSI
The following methods transfer data directly via the [login nodes](../01-Quick-Start-Guide/accessing-interactive-nodes.md#login-nodes-hardware-description). They are suitable for use from **within the PSI network**.
### Rsync (Recommended for Linux/macOS)
### Secure FTP
A `vsftpd` service is available on the login nodes, providing high-speed transfers. Choose the server based on your **speed vs. encryption** needs:
* **`login001.merlin7.psi.ch`:** Encrypted control & data channels.
**Use if your data is sensitive**. **Slower**, but secure.
* **`service03.merlin7.psi.ch`**: Encrypted control channel only.
Use if your data can be transferred unencrypted. **Fastest** method.
!!! tip
The **control channel** is always **encrypted**, therefore, authentication is encrypted and secured.
### Rsync
Rsync is the **preferred** method for small datasets from Linux/macOS systems. It supports **resuming interrupted transfers** and **skips already transferred files**. Syntax:
@@ -51,7 +63,7 @@ rsync -avAHXS ~/localdata $USER@login001.merlin7.psi.ch:/data/project/general/my
```
!!! tip
If a transfer is interrupted, just rerun the command: `rsync` will skip existing files.
If a transfer is interrupted, just rerun the command: `rsync` will skip transferring existing files.
!!! warning
Rsync uses SSH (port 22). For large datasets, transfer speed might be limited.
@@ -64,19 +76,71 @@ SCP works similarly to `rsync` but **does not support resuming** interrupted tra
scp ~/localfile.txt $USER@login001.merlin7.psi.ch:/data/project/general/myproject/
```
### Secure FTP
## Transferring data from outside PSI
A `vsftpd` service is available on the login nodes, providing high-speed transfers. Choose the server based on your **speed vs. encryption** needs:
From August 2024, Merlin is connected to the **[PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)** service,
`datatransfer.psi.ch`. This is a central service managed by the **[Linux team](https://linux.psi.ch/index.html)**. However, any problems or questions related to it can be directly
[reported](../../support/index.md) to the Merlin administrators, which will forward the request if necessary.
* **`login001.merlin7.psi.ch`:** Encrypted control & data channels.
**Use if your data is sensitive**. **Slower**, but secure.
* **`service03.merlin7.psi.ch`**: Encrypted control channel only.
Use if your data can be transferred unencrypted. **Fastest** method.
Access to the PSI Data Transfer services uses ***Multi factor authentication*** (MFA).
Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa).
The PSI Data Transfer servers supports the following protocols:
* Data Transfer `datatransfer.psi.ch` via SSH (scp / rsync / FileZilla)
* Notice that `datatransfer.psi.ch` does not allow SSH login, only `rsync`, `scp` are possible.
* Data Transfer - [www.globus.org](https://www.globus.org/)
* Requires authentication with Switch, as well as a second authentication with PSI.
!!! tip
The **control channel** is always **encrypted**, therefore, authentication is encrypted and secured.
Please follow the [Official PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)
documentation for further instructions.
## UI-based Clients for Data Transfer
!!! warning
When using [Globus](https://www.globus.org), Multi-Factor Authentication (MFA) works only if
**push notifications** are configured as the **only** MFA method in the PSI MFA settings.
If any other method is enabled, such as SMS codes or a combination of _push notifications_ and
_authentication token_, Globus may become unusable.
To fix this, go to your [Microsoft Account Security Information](https://mysignins.microsoft.com/security-info)
and change the sign-in method to **`App Based Authentication - notification`** only.
### FileZilla: datatransfer.psi.ch
One can simply use the FileZilla to transfer data from/to outside PSI. It's important to setup MFA for the connection.
To do this, one has to go to the menu **File -> Site Manager** and create a new endpoint (new site), as follows:
![FileZilla: Connect with MFA](../../images/FileZilla/FileZilla_Connect.png)
It's important that **Logon Type** has the **`interactive`** option set, and the transfer **Protocol** is **`SFTP - SSH File Transfer Protocol`**.
### Connecting to Merlin7 from outside PSI
Merlin7 is fully accessible from within the PSI network. To connect from outside you can use:
* [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN))
* [SSH hopx](https://www.psi.ch/en/computing/ssh-hop)
* Please avoid transferring big amount data through **hop**
* [No Machine](nomachine.md)
* Remote Interactive Access through [**'nx.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access)
* Please avoid transferring big amount of data through **NoMachine**
### Connecting from Merlin7 to outside file shares
#### `merlin_rmount` command
Merlin provides a command for mounting remote file systems, called `merlin_rmount`. This
provides a helpful wrapper over the Gnome storage utilities, and provides support for a wide range of remote file formats, including
* SMB/CIFS (Windows shared folders)
* WebDav
* AFP
* FTP, SFTP
* [others](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8#gvfs-back-ends_managing-storage-volumes-in-gnome)
[More instruction on using `merlin_rmount`](merlin-rmount.md)
## Recommended Data Transfer GUI clients
### WinSCP (Windows)
@@ -102,7 +166,7 @@ Download from [FileZilla Project](https://filezilla-project.org/), or install fr
* `service03.merlin7.psi.ch`: **Fastest** transfer rates, but **data channel not encrypted**.
* Supports drag-and-drop file transfers.
## Sharing Files with SWITCHfilesender
### Sharing Files with SWITCHfilesender
**[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is a Swiss-hosted installation of the [FileSender](https://filesender.org/) project — a web-based application that allows authenticated users to securely and easily send **arbitrarily large files** to other users. Features:
@@ -130,57 +194,3 @@ The service is designed to **send large files for temporary availability**, not
!!! warning
SWITCHfilesender **is not** a long-term storage or archiving solution.
## PSI Data Transfer
From August 2024, Merlin is connected to the **[PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)** service,
`datatransfer.psi.ch`. This is a central service managed by the **[Linux team](https://linux.psi.ch/index.html)**. However, any problems or questions related to it can be directly
[reported](../../support/index.md) to the Merlin administrators, which will forward the request if necessary.
The PSI Data Transfer servers supports the following protocols:
* Data Transfer - SSH (scp / rsync / FileZilla)
* Data Transfer - Globus
Notice that `datatransfer.psi.ch` does not allow SSH login, only `rsync`, `scp` and [Globus](https://www.globus.org/) access is allowed.
Access to the PSI Data Transfer uses ***Multi factor authentication*** (MFA).
Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa).
!!! tip
Please follow the [Official PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)
documentation for further instructions.
### FileZilla example
One can simply use the FileZilla to transfer data from/to outside PSI. It's important to setup MFA for the connection.
To do this, one has to go to the menu **File -> Site Manager** and create a new endpoint (new site), as follows:
![FileZilla: Connect with MFA](../../images/FileZilla/FileZilla_Connect.png)
It's important that **Logon Type** has the **`interactive`** option set, and the transfer **Protocol** is **`SFTP - SSH File Transfer Protocol`**.
## Connecting to Merlin7 from outside PSI
Merlin7 is fully accessible from within the PSI network. To connect from outside you can use:
* [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN))
* [SSH hopx](https://www.psi.ch/en/computing/ssh-hop)
* Please avoid transferring big amount data through **hop**
* [No Machine](nomachine.md)
* Remote Interactive Access through [**'nx.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access)
* Please avoid transferring big amount of data through **NoMachine**
## Connecting from Merlin7 to outside file shares
### `merlin_rmount` command
Merlin provides a command for mounting remote file systems, called `merlin_rmount`. This
provides a helpful wrapper over the Gnome storage utilities, and provides support for a wide range of remote file formats, including
* SMB/CIFS (Windows shared folders)
* WebDav
* AFP
* FTP, SFTP
* [others](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8#gvfs-back-ends_managing-storage-volumes-in-gnome)
[More instruction on using `merlin_rmount`](merlin-rmount.md)