132 lines
5.8 KiB
Markdown
132 lines
5.8 KiB
Markdown
---
|
|
title: Transferring Data
|
|
#tags:
|
|
#keywords:
|
|
last_updated: 9 July 2019
|
|
#summary: ""
|
|
sidebar: merlin6_sidebar
|
|
permalink: /merlin6/transfer-data.html
|
|
---
|
|
|
|
## Transferring Data from the PSI Network to/from Merlin6
|
|
|
|
### Rsync
|
|
|
|
Rsync is the preferred method to transfer data from Linux/MacOS. It allows
|
|
transfers to be easily resumed if they get interrupted. The general syntax is:
|
|
|
|
```
|
|
rsync -avAHXS <src> <dst>
|
|
```
|
|
|
|
For example, to transfer files from your local computer to a merlin project
|
|
directory:
|
|
|
|
```
|
|
rsync -avAHXS ~/localdata user@merlin-l-01.psi.ch:/data/project/general/myproject/
|
|
```
|
|
|
|
You can resume interrupted transfers by simply rerunning the command. Previously
|
|
transferred files will be skipped.
|
|
|
|
|
|
### WinSCP
|
|
|
|
The WinSCP tool can be used for remote file transfer on Windows. It is available
|
|
from the Software Kiosk on PSI machines. Add `merlin-l-01.psi.ch` as a host and
|
|
connect with your PSI credentials. You can then drag-and-drop files between your
|
|
local computer and merlin.
|
|
|
|
## Transferring Data to/from outside PSI
|
|
|
|
Two servers are enabled for exporting data from Merlin to outside PSI.
|
|
These Remote Access Merlin servers are the following:
|
|
* **'ra-merlin-01.psi.ch'**: standard password authentication (with PSI password)
|
|
* `/data/user` mounted in RO (read-only)
|
|
* `/export` directory in RW (read-write). `/export` is also visible from login nodes.
|
|
* **'ra-merlin-02.psi.ch'**: ***Two factor authentication*** (2FA), required **RSA SecurID** token (same as VPN)
|
|
* `/data/project` directories mounted in RW on demand. Project responsibles must request it.
|
|
* `/data/user` mounted in RW (read-write)
|
|
* `/export` directory in RW (read-write). `/export` is also visible from login nodes.
|
|
|
|
In the future, **'ra-merlin-01.psi.ch'** will be also configured with 2FA and will mount the same
|
|
as **'ra-merlin-02.psi.ch'**. In the meantime, we keep **'ra-merlin-01.psi.ch'** with standard authentication
|
|
until we can ensure that most of the Merlin users have a RSA SecurID token or until PSI security policy makes
|
|
its use mandatory. Using **'ra-merlin-02.psi.ch'** over **'ra-merlin-01.psi.ch'** is always recommended (2FA
|
|
is always more secure than standard authentication)
|
|
|
|
### Directories
|
|
|
|
#### /data/user
|
|
|
|
User data directories are mounted in RO on 'ra-merlin-01', and RW on 'ra-merlin-02'.
|
|
|
|
{{site.data.alerts.warning}}Please, <b>ensure proper secured permissions</b> in your '/data/user'
|
|
directory. By default, when directory is created, the system applies the most restrictive
|
|
permissions. However, this does not prevent users for changing permissions if they wish. At this
|
|
point, users become responsible of those changes.
|
|
{{site.data.alerts.end}}
|
|
|
|
#### /export
|
|
|
|
Transferring big amounts of data from outside PSI to Merlin is always possible through `/export`.
|
|
|
|
{{site.data.alerts.tip}}<b>The '/export' directory can be used by any Merlin user.</b>
|
|
This is configured in Read/Write mode. If you need access, please, contact the Merlin administrators.
|
|
{{site.data.alerts.end}}
|
|
|
|
{{site.data.alerts.warning}}The use <b>export</b> as an extension of the quota <i>is forbidden</i>.
|
|
<br><b><i>Auto cleanup policies</i></b> in the <b>export</b> area apply for files older than 28 days.
|
|
{{site.data.alerts.end}}
|
|
|
|
##### Exporting data from Merlin
|
|
|
|
For exporting data from Merlin to outside PSI by using `/export`, one has to:
|
|
* From a Merlin login node, copy your data from any directory (i.e. `/data/project`, `/data/user`, `/scratch`) to
|
|
`/export`. Ensure to properly secure your directories and files with proper permissions.
|
|
* Once data is copied, from **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from `/export` to outside PSI.
|
|
|
|
##### Importing data to Merlin
|
|
|
|
For importing data from outside PSI to Merlin by using `/export`, one has to:
|
|
* From **ra-merlin-01.psi.ch** or **ra-merlin-02.psi.ch**, copy the data from outside PSI to `/export`.
|
|
Ensure to properly secure your directories and files with proper permissions.
|
|
* Once data is copied, from a Merlin login node, copy your data from `/export` to any directory (i.e. `/data/project`, `/data/user`, `/scratch`).
|
|
|
|
#### /data/project
|
|
|
|
Optionally, instead of using `/export`, experiments with a Merlin project can request Read/Write or Read/Only access to their project directory.
|
|
|
|
{{site.data.alerts.tip}}<b>Merlin projects can request direct access on 'ra-merlin-02.psi.ch'</b>
|
|
This can be configured in Read/Write or Read/Only modes. If your project needs access, please,
|
|
contact the Merlin administrators.
|
|
{{site.data.alerts.end}}
|
|
|
|
### Accepted protocols
|
|
|
|
Accepted protocols for Remote Access Merlin servers are the following:
|
|
* **sftp**: **``sftp``** command or similar X11/Windows/MacOS based programs.
|
|
* **ssh**: **`scp`** command (as well as **WinSCP** and similar programs) or **`rsync`** command
|
|
* **~~Globus Online~~**: ***not available yet.***
|
|
|
|
### Remote Access Servers Policies
|
|
|
|
SSH is one of the allowed protocols.
|
|
* Please, **absolutely never** use this servers as a login node.
|
|
* Please avoid copying files to the *home* directories.
|
|
* Please **never use SSH Keys** for accessing these servers. Accessing through SSH keys will be denied in the upcomig months.
|
|
|
|
Only ``/data/user`, `/data/project` and `/export` directories should be used on these nodes,
|
|
and exclusively for transferring data from/to PSI to/from outside PSI.
|
|
|
|
## Connecting to Merlin6 from outside PSI
|
|
|
|
Merlin6 is fully accessible from within the PSI network. To connect from outside you can use:
|
|
|
|
- [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN))
|
|
- [SSH hop](https://www.psi.ch/en/computing/ssh-hop)
|
|
* Please avoid transferring big amount data through **hop**
|
|
- [No Machine](nomachine.md)
|
|
* Remote Interactive Access through [**'rem-acc.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access)
|
|
* Please avoid transferring big amount of data through **NoMachine**
|