diff --git a/README.md b/README.md index 86cef4b..d7e2861 100644 --- a/README.md +++ b/README.md @@ -48,12 +48,14 @@ by Tom Johnson. title: Introduction sidebar: merlin6_sidebar permalink: /merlin6/introduction.html + keywords: key1, key2 --- ``` - Sidebars are specified in data files, e.g. `_data/sidebars/merlin6_sidebar.yml`. - The top menu is controlled by `_data/topnav.yml` - News can be addin in `_posts`. Filenames must include the date. - Lots of features still need to be configured (e.g. pdf output, tags, etc) +- The search bar uses finds substring of the title, tags, keywords, and summary frontmatter. ## License diff --git a/_data/sidebars/merlin6_sidebar.yml b/_data/sidebars/merlin6_sidebar.yml index 8d64445..ac16383 100644 --- a/_data/sidebars/merlin6_sidebar.yml +++ b/_data/sidebars/merlin6_sidebar.yml @@ -103,6 +103,8 @@ entries: url: /merlin6/ansys-hfss.html - title: GOTHIC url: /merlin6/gothic.html + - title: merlin_rmount + url: /merlin6/merlin-rmount.html - title: IntelMPI url: /merlin6/impi.html - title: OpenMPI diff --git a/images/rmount/mount.png b/images/rmount/mount.png new file mode 100644 index 0000000..27cea95 Binary files /dev/null and b/images/rmount/mount.png differ diff --git a/images/rmount/select-mount.png b/images/rmount/select-mount.png new file mode 100644 index 0000000..59d90b9 Binary files /dev/null and b/images/rmount/select-mount.png differ diff --git a/images/rmount/thunar_mount.png b/images/rmount/thunar_mount.png new file mode 100644 index 0000000..a970961 Binary files /dev/null and b/images/rmount/thunar_mount.png differ diff --git a/pages/merlin6/02-How-To-Use-Merlin/merlin-rmount.md b/pages/merlin6/02-How-To-Use-Merlin/merlin-rmount.md new file mode 100644 index 0000000..1c42496 --- /dev/null +++ b/pages/merlin6/02-How-To-Use-Merlin/merlin-rmount.md @@ -0,0 +1,109 @@ +--- +title: Using merlin_rmount +#tags: +keywords: >- + transferring data, data transfer, rsync, dav, webdav, sftp, ftp, smb, cifs, + copy data, copying, mount, file, folder, sharing +last_updated: 24 August 2023 +#summary: "" +sidebar: merlin6_sidebar +permalink: /merlin6/merlin-rmount.html +--- + +## Background + +Merlin provides a command for mounting remote file systems, called `merlin_rmount`. This +provides a helpful wrapper over the Gnome storage utilities (GIO and GVFS), and provides support for a wide range of remote file formats, including +- SMB/CIFS (Windows shared folders) +- WebDav +- AFP +- FTP, SFTP +- [complete list](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8#gvfs-back-ends_managing-storage-volumes-in-gnome) + + +## Usage + + +### Start a session + +First, start a new session. This will start a new bash shell in the current terminal where you can add further commands. + +``` +$ merlin_rmount --init +[INFO] Starting new D-Bus RMOUNT session + +(RMOUNT STARTED) [bliven_s@merlin-l-002 ~]$ +``` + +Note that behind the scenes this is creating a new dbus daemon. Running multiple daemons on the same login node leads to unpredictable results, so it is best not to initialize multiple sessions in parallel. + +### Standard Endpoints + +Standard endpoints can be mounted using + +``` +merlin_rmount --select-mount +``` + +Select the desired url using the arrow keys. + +![merlin_rmount --select-mount](/images/rmount/select-mount.png) + +From this list any of the standard supported endpoints can be mounted. + +### Other endpoints + +Other endpoints can be mounted using the `merlin_rmount --mount ` command. + +![merlin_rmount --mount](/images/rmount/mount.png) + + +### Accessing Files + +After mounting a volume the script will print the mountpoint. It should be of the form + +``` +/run/user/$UID/gvfs/ +``` + +where `$UID` gives your unix user id (a 5-digit number, also viewable with `id -u`) and +`` is some string generated from the mount options. + +For convenience, it may be useful to add a symbolic link for this gvfs directory. For instance, this would allow all volumes to be accessed in ~/mnt/: + +``` +ln -s ~/mnt /run/user/$UID/gvfs +``` + +Files are accessible as long as the `merlin_rmount` shell remains open. + + +### Disconnecting + +To disconnect, close the session with one of the following: + +- The exit command +- CTRL-D +- Closing the terminal + +Disconnecting will unmount all volumes. + + +## Alternatives + +### Thunar + +Users that prefer a GUI file browser may prefer the `thunar` command, which opens the Gnome File Browser. This is also available in NoMachine sessions in the bottom bar (1). Thunar supports the same remote filesystems as `merlin_rmount`; just type the URL in the address bar (2). + +![Mounting with thunar](/images/rmount/thunar_mount.png) + +When using thunar within a NoMachine session, file transfers continue after closing NoMachine (as long as the NoMachine session stays active). + +Files can also be accessed at the command line as needed (see 'Accessing Files' above). + +## Resources + +- [BIO docs](https://intranet.psi.ch/en/bio/webdav-data) on using these tools for + transfering EM data +- [Redhad docs on GVFS](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8) +- [gio reference](https://developer-old.gnome.org/gio/stable/gio.html) diff --git a/pages/merlin6/02-How-To-Use-Merlin/transfer-data.md b/pages/merlin6/02-How-To-Use-Merlin/transfer-data.md index a5c027e..97a6bbd 100644 --- a/pages/merlin6/02-How-To-Use-Merlin/transfer-data.md +++ b/pages/merlin6/02-How-To-Use-Merlin/transfer-data.md @@ -2,13 +2,34 @@ title: Transferring Data #tags: keywords: transferring data, data transfer, rsync, winscp, copy data, copying, sftp, import, export, hop, vpn -last_updated: 07 September 2022 +last_updated: 24 August 2023 #summary: "" sidebar: merlin6_sidebar permalink: /merlin6/transfer-data.html --- -## Transferring Data from the PSI Network to/from Merlin6 +## Overview + +Most methods allow data to be either transmitted or received, so it may make sense to +initiate the transfer from either merlin or the other system, depending on the network +visibility. + +- Merlin login nodes are visible from the PSI network, so direct data transfer + (rsync/WinSCP) is generally preferable. This can be initiated from either endpoint. +- Merlin login nodes can access the internet using a limited set of protocols + - SSH-based protocols using port 22 (rsync-over-ssh, sftp, WinSCP, etc) + - HTTP-based protocols using ports 80 or 445 (https, WebDav, etc) + - Protocols using other ports require admin configuration and may only work with + specific hosts (ftp, rsync daemons, etc) +- Systems on the internet can access the Remote Access Merlin servers + (ra-merlin*.psi.ch) using ssh-based protocols + + +## Direct transfer via Merlin6 login nodes + +The following methods transfer data directly via the [login +nodes](/merlin6/interactive.html#login-nodes-hardware-description). They are suitable +for use from within the PSI network. ### Rsync @@ -37,9 +58,10 @@ from the Software Kiosk on PSI machines. Add `merlin-l-01.psi.ch` as a host and connect with your PSI credentials. You can then drag-and-drop files between your local computer and merlin. -## Transferring Data to/from outside PSI -Two servers are enabled for exporting data from Merlin outside PSI, as well as for importing data. +## Remote Access Servers + +Two servers are enabled for data transfers originating from outside PSI. This is a central service managed by a different team, which is managing the different Remote Access services at PSI for the different facilities (including the one for Merlin). However, any problems or questions related to it can be directly [reported](/merlin6/contact.html) to the Merlin adminstrators, @@ -132,3 +154,18 @@ Merlin6 is fully accessible from within the PSI network. To connect from outside - [No Machine](nomachine.md) * Remote Interactive Access through [**'rem-acc.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access) * Please avoid transferring big amount of data through **NoMachine** + +## Connecting from Merlin6 to outside file shares + +### `merlin_rmount` command + +Merlin provides a command for mounting remote file systems, called `merlin_rmount`. This +provides a helpful wrapper over the Gnome storage utilities, and provides support for a wide range of remote file formats, including +- SMB/CIFS (Windows shared folders) +- WebDav +- AFP +- FTP, SFTP +- [others](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/managing-storage-volumes-in-gnome_using-the-desktop-environment-in-rhel-8#gvfs-back-ends_managing-storage-volumes-in-gnome) + + +[More instruction on using `merlin_rmount`](/merlin6/merlin-rmount.html)