--- title: Transferring Data #tags: keywords: transferring data, data transfer, rsync, winscp, copy data, copying, sftp, import, export, hop, vpn last_updated: 24 August 2023 #summary: "" sidebar: merlin6_sidebar permalink: /merlin6/transfer-data.html --- ## Overview Most methods allow data to be either transmitted or received, so it may make sense to initiate the transfer from either merlin or the other system, depending on the network visibility. - Merlin login nodes are visible from the PSI network, so direct data transfer (rsync/WinSCP) is generally preferable. This can be initiated from either endpoint. - Merlin login nodes can access the internet using a limited set of protocols - SSH-based protocols using port 22 (rsync-over-ssh, sftp, WinSCP, etc) - HTTP-based protocols using ports 80 or 445 (https, WebDav, etc) - Protocols using other ports require admin configuration and may only work with specific hosts (ftp, rsync daemons, etc) - Systems on the internet can access the Remote Access Merlin servers (ra-merlin*.psi.ch) using ssh-based protocols ## Direct transfer via Merlin6 login nodes The following methods transfer data directly via the [login nodes](/merlin6/interactive.html#login-nodes-hardware-description). They are suitable for use from within the PSI network. ### Rsync Rsync is the preferred method to transfer data from Linux/MacOS. It allows transfers to be easily resumed if they get interrupted. The general syntax is: ``` rsync -avAHXS ``` For example, to transfer files from your local computer to a merlin project directory: ``` rsync -avAHXS ~/localdata user@merlin-l-01.psi.ch:/data/project/general/myproject/ ``` You can resume interrupted transfers by simply rerunning the command. Previously transferred files will be skipped. ### WinSCP The WinSCP tool can be used for remote file transfer on Windows. It is available from the Software Kiosk on PSI machines. Add `merlin-l-01.psi.ch` as a host and connect with your PSI credentials. You can then drag-and-drop files between your local computer and merlin. ## Remote Access Servers Two servers are enabled for data transfers originating from outside PSI. This is a central service managed by a different team, which is managing the different Remote Access services at PSI for the different facilities (including the one for Merlin). However, any problems or questions related to it can be directly [reported](/merlin6/contact.html) to the Merlin adminstrators, which will forward the request if necessary. These Remote Access Merlin servers are the following: * **`ra-merlin-01.psi.ch`** * **`ra-merlin-02.psi.ch`** Both servers have mounted the following Merlin filesystems: * `/data/project` directories mounted in RW on demand. Project responsibles must request it. * `/data/user` mounted in RW (read-write) * `/data/experiment/mu3e` directories mounted in RW (read-write), except `data` (read-only mounted) * `/export` directory in RW (read-write). `/export` is also visible from login nodes. Access to the Remote Access server uses ***Multi factor authentication*** (MFA). Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa). ### Directories #### /data/user User data directories are mounted in RW on both 'ra-merlin-01' and 'ra-merlin-02'. {{site.data.alerts.warning}}Please, ensure proper secured permissions in your '/data/user' directory. By default, when directory is created, the system applies the most restrictive permissions. However, this does not prevent users for changing permissions if they wish. At this point, users become responsible of those changes. {{site.data.alerts.end}} #### /export Transferring big amounts of data from outside PSI to Merlin is always possible through `/export`. {{site.data.alerts.tip}}The '/export' directory can be used by any Merlin user. This is configured in Read/Write mode. If you need access, please, contact the Merlin administrators. {{site.data.alerts.end}} {{site.data.alerts.warning}}The use export as an extension of the quota is forbidden.
Auto cleanup policies in the export area apply for files older than 28 days. {{site.data.alerts.end}} ##### Exporting data from Merlin For exporting data from Merlin to outside PSI by using `/export`, one has to: * From a Merlin login node, copy your data from any directory (i.e. `/data/project`, `/data/user`, `/scratch`) to `/export`. Ensure to properly secure your directories and files with proper permissions. * Once data is copied, from **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from `/export` to outside PSI. ##### Importing data to Merlin For importing data from outside PSI to Merlin by using `/export`, one has to: * From **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from outside PSI to `/export`. Ensure to properly secure your directories and files with proper permissions. * Once data is copied, from a Merlin login node, copy your data from `/export` to any directory (i.e. `/data/project`, `/data/user`, `/scratch`). #### /data/project Optionally, instead of using `/export`, experiments with a Merlin project can request Read/Write or Read/Only access to their project directory. {{site.data.alerts.tip}}Merlin projects can request direct access. This can be configured in Read/Write or Read/Only modes. If your project needs access, please, contact the Merlin administrators. {{site.data.alerts.end}} ### Accepted protocols Accepted protocols for Remote Access Merlin servers are the following: * **sftp**: **``sftp``** command or similar X11/Windows/MacOS based programs. * **ssh**: **`scp`** command (as well as **WinSCP** and similar programs) or **`rsync`** command * **~~Globus Online~~**: ***not available yet.*** ### Remote Access Servers Policies SSH is one of the allowed protocols. * Please, **absolutely never** use this servers as a login node. * Please avoid copying files to the *home* directories. * Please **never use SSH Keys** for accessing these servers. Accessing through SSH keys will be denied in the upcomig months. Only ``/data/user`, `/data/project` and `/export` directories should be used on these nodes, and exclusively for transferring data from/to PSI to/from outside PSI. ## Connecting to Merlin6 from outside PSI Merlin6 is fully accessible from within the PSI network. To connect from outside you can use: - [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN)) - [SSH hop](https://www.psi.ch/en/computing/ssh-hop) * Please avoid transferring big amount data through **hop** - [No Machine](nomachine.md) * Remote Interactive Access through [**'rem-acc.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access) * Please avoid transferring big amount of data through **NoMachine** ## Connecting from Merlin6 to outside file shares ### `merlin_rmount` command Merlin provides a command for mounting remote file systems, called `merlin_rmount`. This provides a helpful wrapper over the Gnome storage utilities, and provides support for a wide range of remote file formats, including - SMB/CIFS (Windows shared folders) - WebDav - AFP - FTP, SFTP - [others](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8#gvfs-back-ends_managing-storage-volumes-in-gnome) [More instruction on using `merlin_rmount`](/merlin6/merlin-rmount.html)