---
title: Transferring Data
#tags:
#keywords:
last_updated: 9 July 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/transfer-data.html
---

## Transferring Data from the PSI Network to/from Merlin6

### Rsync

Rsync is the preferred method to transfer data from Linux/MacOS. It allows
transfers to be easily resumed if they get interrupted. The general syntax is:

```
rsync -avAHXS <src> <dst>
```

For example, to transfer files from your local computer to a merlin project
directory:

```
rsync -avAHXS ~/localdata user@merlin-l-01.psi.ch:/data/project/general/myproject/
```

You can resume interrupted transfers by simply rerunning the command. Previously
transferred files will be skipped.


### WinSCP

The WinSCP tool can be used for remote file transfer on Windows. It is available
from the Software Kiosk on PSI machines. Add `merlin-l-01.psi.ch` as a host and
connect with your PSI credentials. You can then drag-and-drop files between your
local computer and merlin.

## Transferring Data to/from outside PSI

Two servers are enabled for exporting data from Merlin to outside PSI. 
These Remote Access Merlin servers are the following:
* **'ra-merlin-01.psi.ch'**: standard password authentication (with PSI password)
   * `/data/user` mounted in RO (read-only)
   * `/export` directory in RW (read-write). `/export` is also visible from login nodes.
* **'ra-merlin-02.psi.ch'**: ***Two factor authentication*** (2FA), required **RSA SecurID** token (same as VPN)
   * `/data/project` directories mounted in RW on demand. Project responsibles must request it.
   * `/data/user` mounted in RW (read-write)
   * `/export` directory in RW (read-write). `/export` is also visible from login nodes.

In the future, **'ra-merlin-01.psi.ch'** will be also configured with 2FA and will mount the same 
as **'ra-merlin-02.psi.ch'**. In the meantime, we keep **'ra-merlin-01.psi.ch'** with standard authentication 
until we can ensure that most of the Merlin users have a RSA SecurID token or until PSI security policy makes
its use mandatory. Using **'ra-merlin-02.psi.ch'** over **'ra-merlin-01.psi.ch'** is always recommended (2FA
is always more secure than standard authentication)

### Directories

#### /data/user

User data directories are mounted in RO on 'ra-merlin-01', and RW on 'ra-merlin-02'.

{{site.data.alerts.warning}}Please, <b>ensure proper secured permissions</b> in your '/data/user' 
directory. By default, when directory is created, the system applies the most restrictive 
permissions. However, this does not prevent users for changing permissions if they wish. At this 
point, users become responsible of those changes.
{{site.data.alerts.end}}

#### /export

Transferring big amounts of data from outside PSI to Merlin is always possible through `/export`.

{{site.data.alerts.tip}}<b>The '/export' directory can be used by any Merlin user.</b>
This is configured in Read/Write mode. If you need access, please, contact the Merlin administrators.
{{site.data.alerts.end}}

{{site.data.alerts.warning}}The use <b>export</b> as an extension of the quota <i>is forbidden</i>. 
<br><b><i>Auto cleanup policies</i></b> in the <b>export</b> area apply for files older than 28 days.
{{site.data.alerts.end}}

##### Exporting data from Merlin

For exporting data from Merlin to outside PSI by using `/export`, one has to:
   * From a Merlin login node, copy your data from any directory (i.e. `/data/project`, `/data/user`, `/scratch`) to
`/export`. Ensure to properly secure your directories and files with proper permissions.
   * Once data is copied, from **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from `/export` to outside PSI.

##### Importing data to Merlin

For importing data from outside PSI to Merlin by using `/export`, one has to:
   * From **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from outside PSI to `/export`.
Ensure to properly secure your directories and files with proper permissions.
   * Once data is copied, from a Merlin login node, copy your data from `/export` to any directory (i.e. `/data/project`, `/data/user`, `/scratch`). 

#### /data/project

Optionally, instead of using `/export`, experiments with a Merlin project can request Read/Write or Read/Only access to their project directory.

{{site.data.alerts.tip}}<b>Merlin projects can request direct access on 'ra-merlin-02.psi.ch'</b>
This can be configured in Read/Write or Read/Only modes. If your project needs access, please,
contact the Merlin administrators.
{{site.data.alerts.end}}

### Accepted protocols

Accepted protocols for Remote Access Merlin servers are the following:
* **sftp**: **``sftp``** command or similar X11/Windows/MacOS based programs.
* **ssh**: **`scp`** command (as well as **WinSCP** and similar programs) or **`rsync`** command
* **~~Globus Online~~**: ***not available yet.***

### Remote Access Servers Policies

SSH is one of the allowed protocols. 
* Please, **absolutely never** use this servers as a login node.
* Please avoid copying files to the *home* directories. 
* Please **never use SSH Keys** for accessing these servers. Accessing through SSH keys will be denied in the upcomig months.

Only ``/data/user`, `/data/project` and `/export` directories should be used on these nodes, 
and exclusively for transferring data from/to PSI to/from outside PSI.

## Connecting to Merlin6 from outside PSI

Merlin6 is fully accessible from within the PSI network. To connect from outside you can use:

- [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN))
- [SSH hop](https://www.psi.ch/en/computing/ssh-hop)
   * Please avoid transferring big amount data through **hop**
- [No Machine](nomachine.md)
   * Remote Interactive Access through [**'rem-acc.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access)
   * Please avoid transferring big amount of data through **NoMachine**
