first stab at mkdocs migration
refactor CSCS and Meg content add merlin6 quick start update merlin6 nomachine docs give the userdoc its own color scheme we use the Materials default one refactored slurm general docs merlin6 add merlin6 JB docs add software support m6 docs add all files to nav vibed changes #1 add missing pages further vibing #2 vibe #3 further fixes
This commit is contained in:
165
docs/merlin6/how-to-use-merlin/transfer-data.md
Normal file
165
docs/merlin6/how-to-use-merlin/transfer-data.md
Normal file
@@ -0,0 +1,165 @@
|
||||
---
|
||||
title: Transferring Data
|
||||
#tags:
|
||||
keywords: transferring data, data transfer, rsync, winscp, copy data, copying, sftp, import, export, hopx, vpn
|
||||
last_updated: 24 August 2023
|
||||
#summary: ""
|
||||
sidebar: merlin6_sidebar
|
||||
permalink: /merlin6/transfer-data.html
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Most methods allow data to be either transmitted or received, so it may make sense to
|
||||
initiate the transfer from either merlin or the other system, depending on the network
|
||||
visibility.
|
||||
|
||||
- Merlin login nodes are visible from the PSI network, so direct data transfer
|
||||
(rsync/WinSCP) is generally preferable. This can be initiated from either endpoint.
|
||||
- Merlin login nodes can access the internet using a limited set of protocols
|
||||
- SSH-based protocols using port 22 (rsync-over-ssh, sftp, WinSCP, etc)
|
||||
- HTTP-based protocols using ports 80 or 445 (https, WebDav, etc)
|
||||
- Protocols using other ports require admin configuration and may only work with
|
||||
specific hosts (ftp, rsync daemons, etc)
|
||||
- Systems on the internet can access the [PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer) service
|
||||
`datatransfer.psi.ch`, using ssh-based protocols and [Globus](https://www.globus.org/)
|
||||
|
||||
|
||||
## Direct transfer via Merlin6 login nodes
|
||||
|
||||
The following methods transfer data directly via the [login
|
||||
nodes](../quick-start-guide/accessing-interactive-nodes.md). They are suitable
|
||||
for use from within the PSI network.
|
||||
|
||||
### Rsync
|
||||
|
||||
Rsync is the preferred method to transfer data from Linux/MacOS. It allows
|
||||
transfers to be easily resumed if they get interrupted. The general syntax is:
|
||||
|
||||
```
|
||||
rsync -avAHXS <src> <dst>
|
||||
```
|
||||
|
||||
For example, to transfer files from your local computer to a merlin project
|
||||
directory:
|
||||
|
||||
```
|
||||
rsync -avAHXS ~/localdata user@merlin-l-01.psi.ch:/data/project/general/myproject/
|
||||
```
|
||||
|
||||
You can resume interrupted transfers by simply rerunning the command. Previously
|
||||
transferred files will be skipped.
|
||||
|
||||
|
||||
### WinSCP
|
||||
|
||||
The WinSCP tool can be used for remote file transfer on Windows. It is available
|
||||
from the Software Kiosk on PSI machines. Add `merlin-l-01.psi.ch` as a host and
|
||||
connect with your PSI credentials. You can then drag-and-drop files between your
|
||||
local computer and merlin.
|
||||
|
||||
### SWITCHfilesender
|
||||
|
||||
**[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is an installation of the FileSender project (filesender.org) which is a web based application that allows authenticated users to securely and easily send arbitrarily large files to other users.
|
||||
|
||||
Authentication of users is provided through SimpleSAMLphp, supporting SAML2, LDAP and RADIUS and more. Users without an account can be sent an upload voucher by an authenticated user. FileSender is developed to the requirements of the higher education and research community.
|
||||
|
||||
The purpose of the software is to send a large file to someone, have that file available for download for a certain number of downloads and/or a certain amount of time, and after that automatically delete the file. The software is not intended as a permanent file publishing platform.
|
||||
|
||||
**[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is fully integrated with PSI, therefore, PSI employees can log in by using their PSI account (through Authentication and Authorization Infrastructure / AAI, by selecting PSI as the institution to be used for log in).
|
||||
|
||||
## PSI Data Transfer
|
||||
|
||||
From August 2024, Merlin is connected to the **[PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)** service,
|
||||
`datatransfer.psi.ch`. This is a central service managed by the **[Linux team](https://linux.psi.ch/index.html)**. However, any problems or questions related to it can be directly
|
||||
[reported](../99-support/contact.md) to the Merlin administrators, which will forward the request if necessary.
|
||||
|
||||
The PSI Data Transfer servers supports the following protocols:
|
||||
* Data Transfer - SSH (scp / rsync)
|
||||
* Data Transfer - Globus
|
||||
|
||||
Notice that `datatransfer.psi.ch` does not allow SSH login, only `rsync`, `scp` and [Globus](https://www.globus.org/) access is allowed.
|
||||
|
||||
The following filesystems are mounted:
|
||||
* `/merlin/export` which points to the `/export` directory in Merlin.
|
||||
* `/merlin/data/experiment/mu3e` which points to the `/data/experiment/mu3e` directories in Merlin.
|
||||
* Mu3e sub-directories are mounted in RW (read-write), except for `data` (read-only mounted)
|
||||
* `/merlin/data/project/general` which points to the `/data/project/general` directories in Merlin.
|
||||
* Owners of Merlin projects should request explicit access to it.
|
||||
* Currently, only `CSCS` is available for transferring files between PizDaint/Alps and Merlin
|
||||
* `/merlin/data/project/bio` which points to the `/data/project/bio` directories in Merlin.
|
||||
* `/merlin/data/user` which points to the `/data/user` directories in Merlin.
|
||||
|
||||
Access to the PSI Data Transfer uses ***Multi factor authentication*** (MFA).
|
||||
Therefore, having the Microsoft Authenticator App is required as explained [here](https://www.psi.ch/en/computing/change-to-mfa).
|
||||
|
||||
!!! tip "Official Documentation"
|
||||
Please follow the [Official PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer) documentation for further instructions.
|
||||
|
||||
### Directories
|
||||
|
||||
#### /merlin/data/user
|
||||
|
||||
User data directories are mounted in RW.
|
||||
|
||||
!!! warning "Secure Permissions"
|
||||
Please, **ensure proper secured permissions** in your `/data/user` directory. By default, when directory is created, the system applies the most restrictive permissions. However, this does not prevent users for changing permissions if they wish. At this point, users become responsible of those changes.
|
||||
|
||||
#### /merlin/export
|
||||
|
||||
Transferring big amounts of data from outside PSI to Merlin is always possible through `/export`.
|
||||
|
||||
!!! tip "Export Directory Access"
|
||||
The `/export` directory can be used by any Merlin user. This is configured in Read/Write mode. If you need access, please, contact the Merlin administrators.
|
||||
|
||||
!!! warning "Export Usage Policy"
|
||||
The use **export** as an extension of the quota *is forbidden*.
|
||||
|
||||
Auto cleanup policies in the **export** area apply for files older than 28 days.
|
||||
|
||||
##### Exporting data from Merlin
|
||||
|
||||
For exporting data from Merlin to outside PSI by using `/export`, one has to:
|
||||
* From a Merlin login node, copy your data from any directory (i.e. `/data/project`, `/data/user`, `/scratch`) to
|
||||
`/export`. Ensure to properly secure your directories and files with proper permissions.
|
||||
* Once data is copied, from **`datatransfer.psi.ch`**, copy the data from `/merlin/export` to outside PSI
|
||||
|
||||
##### Importing data to Merlin
|
||||
|
||||
For importing data from outside PSI to Merlin by using `/export`, one has to:
|
||||
* From **`datatransfer.psi.ch`**, copy the data from outside PSI to `/merlin/export`.
|
||||
Ensure to properly secure your directories and files with proper permissions.
|
||||
* Once data is copied, from a Merlin login node, copy your data from `/export` to any directory (i.e. `/data/project`, `/data/user`, `/scratch`).
|
||||
|
||||
#### Request access to your project directory
|
||||
|
||||
Optionally, instead of using `/export`, Merlin project owners can request Read/Write or Read/Only access to their project directory.
|
||||
|
||||
!!! tip "Project Access"
|
||||
Merlin projects can request direct access. This can be configured in Read/Write or Read/Only modes. If your project needs access, please, contact the Merlin administrators.
|
||||
|
||||
## Connecting to Merlin6 from outside PSI
|
||||
|
||||
Merlin6 is fully accessible from within the PSI network. To connect from outside you can use:
|
||||
|
||||
- [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN))
|
||||
- [SSH hopx](https://www.psi.ch/en/computing/ssh-hop)
|
||||
* Please avoid transferring big amount data through **hopx**
|
||||
- [No Machine](nomachine.md)
|
||||
* Remote Interactive Access through [**'rem-acc.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access)
|
||||
* Please avoid transferring big amount of data through **NoMachine**
|
||||
|
||||
## Connecting from Merlin6 to outside file shares
|
||||
|
||||
### `merlin_rmount` command
|
||||
|
||||
Merlin provides a command for mounting remote file systems, called `merlin_rmount`. This
|
||||
provides a helpful wrapper over the Gnome storage utilities, and provides support for a wide range of remote file formats, including
|
||||
- SMB/CIFS (Windows shared folders)
|
||||
- WebDav
|
||||
- AFP
|
||||
- FTP, SFTP
|
||||
- [others](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8#gvfs-back-ends_managing-storage-volumes-in-gnome)
|
||||
|
||||
|
||||
[More instruction on using `merlin_rmount`](../software-support/merlin-rmount.md)
|
||||
Reference in New Issue
Block a user