From b0d3dd2f8cb2ac8581429a0d71a8aaf5bf0f9503 Mon Sep 17 00:00:00 2001 From: caubet_m Date: Tue, 31 Mar 2026 15:58:53 +0200 Subject: [PATCH] Improve docs --- .../02-How-To-Use-Merlin/transfer-data.md | 142 ++++++++++-------- 1 file changed, 76 insertions(+), 66 deletions(-) diff --git a/docs/merlin7/02-How-To-Use-Merlin/transfer-data.md b/docs/merlin7/02-How-To-Use-Merlin/transfer-data.md index d95f914e..cb779405 100644 --- a/docs/merlin7/02-How-To-Use-Merlin/transfer-data.md +++ b/docs/merlin7/02-How-To-Use-Merlin/transfer-data.md @@ -32,11 +32,23 @@ Most data transfer methods support both sending and receiving, so you may initia The next chapters contain detailed information about the different transfer methods available on Merlin7. -## Direct Transfer via Merlin7 Login Nodes +## Transferring from within PSI The following methods transfer data directly via the [login nodes](../01-Quick-Start-Guide/accessing-interactive-nodes.md#login-nodes-hardware-description). They are suitable for use from **within the PSI network**. -### Rsync (Recommended for Linux/macOS) +### Secure FTP + +A `vsftpd` service is available on the login nodes, providing high-speed transfers. Choose the server based on your **speed vs. encryption** needs: + +* **`login001.merlin7.psi.ch`:** Encrypted control & data channels. + **Use if your data is sensitive**. **Slower**, but secure. +* **`service03.merlin7.psi.ch`**: Encrypted control channel only. + Use if your data can be transferred unencrypted. **Fastest** method. + +!!! tip + The **control channel** is always **encrypted**, therefore, authentication is encrypted and secured. + +### Rsync Rsync is the **preferred** method for small datasets from Linux/macOS systems. It supports **resuming interrupted transfers** and **skips already transferred files**. Syntax: @@ -51,7 +63,7 @@ rsync -avAHXS ~/localdata $USER@login001.merlin7.psi.ch:/data/project/general/my ``` !!! tip - If a transfer is interrupted, just rerun the command: `rsync` will skip existing files. + If a transfer is interrupted, just rerun the command: `rsync` will skip transferring existing files. !!! warning Rsync uses SSH (port 22). For large datasets, transfer speed might be limited. @@ -64,19 +76,71 @@ SCP works similarly to `rsync` but **does not support resuming** interrupted tra scp ~/localfile.txt $USER@login001.merlin7.psi.ch:/data/project/general/myproject/ ``` -### Secure FTP +## Transferring data from outside PSI -A `vsftpd` service is available on the login nodes, providing high-speed transfers. Choose the server based on your **speed vs. encryption** needs: +From August 2024, Merlin is connected to the **[PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)** service, +`datatransfer.psi.ch`. This is a central service managed by the **[Linux team](https://linux.psi.ch/index.html)**. However, any problems or questions related to it can be directly +[reported](../../support/index.md) to the Merlin administrators, which will forward the request if necessary. -* **`login001.merlin7.psi.ch`:** Encrypted control & data channels. - **Use if your data is sensitive**. **Slower**, but secure. -* **`service03.merlin7.psi.ch`**: Encrypted control channel only. - Use if your data can be transferred unencrypted. **Fastest** method. +Access to the PSI Data Transfer services uses ***Multi factor authentication*** (MFA). +Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa). + +The PSI Data Transfer servers supports the following protocols: +* Data Transfer `datatransfer.psi.ch` via SSH (scp / rsync / FileZilla) + * Notice that `datatransfer.psi.ch` does not allow SSH login, only `rsync`, `scp` are possible. +* Data Transfer - [www.globus.org](https://www.globus.org/) + * Requires authentication with Switch, as well as a second authentication with PSI. !!! tip - The **control channel** is always **encrypted**, therefore, authentication is encrypted and secured. + Please follow the [Official PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer) + documentation for further instructions. -## UI-based Clients for Data Transfer +!!! warning + When using [Globus](https://www.globus.org), Multi-Factor Authentication (MFA) works only if + **push notifications** are configured as the **only** MFA method in the PSI MFA settings. + + If any other method is enabled, such as SMS codes or a combination of _push notifications_ and + _authentication token_, Globus may become unusable. + + To fix this, go to your [Microsoft Account Security Information](https://mysignins.microsoft.com/security-info) + and change the sign-in method to **`App Based Authentication - notification`** only. + +### FileZilla: datatransfer.psi.ch + +One can simply use the FileZilla to transfer data from/to outside PSI. It's important to setup MFA for the connection. +To do this, one has to go to the menu **File -> Site Manager** and create a new endpoint (new site), as follows: + + ![FileZilla: Connect with MFA](../../images/FileZilla/FileZilla_Connect.png) + +It's important that **Logon Type** has the **`interactive`** option set, and the transfer **Protocol** is **`SFTP - SSH File Transfer Protocol`**. + +### Connecting to Merlin7 from outside PSI + +Merlin7 is fully accessible from within the PSI network. To connect from outside you can use: + +* [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN)) +* [SSH hopx](https://www.psi.ch/en/computing/ssh-hop) + * Please avoid transferring big amount data through **hop** +* [No Machine](nomachine.md) + * Remote Interactive Access through [**'nx.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access) + * Please avoid transferring big amount of data through **NoMachine** + +### Connecting from Merlin7 to outside file shares + +#### `merlin_rmount` command + +Merlin provides a command for mounting remote file systems, called `merlin_rmount`. This +provides a helpful wrapper over the Gnome storage utilities, and provides support for a wide range of remote file formats, including + +* SMB/CIFS (Windows shared folders) +* WebDav +* AFP +* FTP, SFTP +* [others](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8#gvfs-back-ends_managing-storage-volumes-in-gnome) + +[More instruction on using `merlin_rmount`](merlin-rmount.md) + +## Recommended Data Transfer GUI clients ### WinSCP (Windows) @@ -102,7 +166,7 @@ Download from [FileZilla Project](https://filezilla-project.org/), or install fr * `service03.merlin7.psi.ch`: **Fastest** transfer rates, but **data channel not encrypted**. * Supports drag-and-drop file transfers. -## Sharing Files with SWITCHfilesender +### Sharing Files with SWITCHfilesender **[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is a Swiss-hosted installation of the [FileSender](https://filesender.org/) project — a web-based application that allows authenticated users to securely and easily send **arbitrarily large files** to other users. Features: @@ -130,57 +194,3 @@ The service is designed to **send large files for temporary availability**, not !!! warning SWITCHfilesender **is not** a long-term storage or archiving solution. -## PSI Data Transfer - -From August 2024, Merlin is connected to the **[PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)** service, -`datatransfer.psi.ch`. This is a central service managed by the **[Linux team](https://linux.psi.ch/index.html)**. However, any problems or questions related to it can be directly -[reported](../../support/index.md) to the Merlin administrators, which will forward the request if necessary. - -The PSI Data Transfer servers supports the following protocols: - -* Data Transfer - SSH (scp / rsync / FileZilla) -* Data Transfer - Globus - -Notice that `datatransfer.psi.ch` does not allow SSH login, only `rsync`, `scp` and [Globus](https://www.globus.org/) access is allowed. - -Access to the PSI Data Transfer uses ***Multi factor authentication*** (MFA). -Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa). - -!!! tip - Please follow the [Official PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer) - documentation for further instructions. - -### FileZilla example - -One can simply use the FileZilla to transfer data from/to outside PSI. It's important to setup MFA for the connection. -To do this, one has to go to the menu **File -> Site Manager** and create a new endpoint (new site), as follows: - - ![FileZilla: Connect with MFA](../../images/FileZilla/FileZilla_Connect.png) - -It's important that **Logon Type** has the **`interactive`** option set, and the transfer **Protocol** is **`SFTP - SSH File Transfer Protocol`**. - -## Connecting to Merlin7 from outside PSI - -Merlin7 is fully accessible from within the PSI network. To connect from outside you can use: - -* [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN)) -* [SSH hopx](https://www.psi.ch/en/computing/ssh-hop) - * Please avoid transferring big amount data through **hop** -* [No Machine](nomachine.md) - * Remote Interactive Access through [**'nx.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access) - * Please avoid transferring big amount of data through **NoMachine** - -## Connecting from Merlin7 to outside file shares - -### `merlin_rmount` command - -Merlin provides a command for mounting remote file systems, called `merlin_rmount`. This -provides a helpful wrapper over the Gnome storage utilities, and provides support for a wide range of remote file formats, including - -* SMB/CIFS (Windows shared folders) -* WebDav -* AFP -* FTP, SFTP -* [others](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8#gvfs-back-ends_managing-storage-volumes-in-gnome) - -[More instruction on using `merlin_rmount`](merlin-rmount.md)