gitea-pages/docs/cls/data_transfer/uni_basel.md

Uni Basel

The Uni Basel data-transfer server is directly accessible from the Merlin 7 login nodes using SFTP.

The examples below show how to automate file listing and data transfer using Bash and expect. The scripts were originally provided by P. Filipcik.

!!! warning "Password security" The examples below use expect to provide the SFTP password automatically. Do not commit scripts containing your password to version control or share them with others. For regular or automated transfers, consider using SSH keys or passing the password through a protected environment variable or other secure mechanism.

In the examples, the username on the Uni Basel SFTP server is sftp. Replace the example password with your own credentials and adjust the paths as needed.

1. Listing files and folders

The following script lists the directories available in the share directory on the Uni Basel server.

#!/bin/bash

# Usage: ./bioem-list.sh

echo "Listing directories in BioEM share"

expect <<EOF
spawn sftp -P 3022 "sftp@131.152.226.6"

expect "password:"
send "your_password\r"

expect "sftp>"
send "ls share/\r"

expect "sftp>"
send "bye\r"
EOF

Save the script as bioem-list.sh and make it executable:

chmod +x bioem-list.sh

Then run:

./bioem-list.sh

2. Retrieving data

The following script downloads a directory from the Uni Basel server.

The directory name should be specified without the share/ prefix. Run the script from the directory where you want the downloaded data to be stored.

#!/bin/bash

# Usage: ./bioem-get.sh <folder-name>
#
# Example:
#   ./bioem-get.sh 2024-09-01_mydata

if [ "$#" -lt 1 ]; then
    echo "Illegal number of parameters"
    echo "Usage: ./bioem-get.sh <folder-name>"
    echo "Example: ./bioem-get.sh 2024-09-01_mydata"
    exit 1
fi

folder="$1"

expect <<EOF
spawn sftp -P 3022 "sftp@131.152.226.6"

expect "password:"
send "your_password\r"

expect "sftp>"
send "get -R share/$folder\r"

expect "sftp>"
send "bye\r"
EOF

Make the script executable:

chmod +x bioem-get.sh

For example, to download the directory 2024-09-01_mydata:

./bioem-get.sh 2024-09-01_mydata

The data will be downloaded to the current working directory.

!!! note "Recursive download" The -R option in the SFTP get command recursively downloads the specified directory and all of its contents.

!!! note "Run from the target directory" Run bioem-get.sh from the directory where you want the downloaded data to be stored.

Password handling

The examples above contain a placeholder for the password: your_password

Replace this with your actual password if you use the scripts as written.

!!! warning "Avoid hardcoding passwords" Hardcoding a password directly in a script is not recommended because anyone who can read the script can also read the password. For repeated or automated transfers, use a more secure authentication method, such as SSH keys, where supported.

If password-based authentication is required, consider using a protected `expect` wrapper or another mechanism that keeps the password outside the script.