3.3 KiB
Nexus ETH Zurich
This guide describes how to automatically transfer data from Nexus (ETH Zurich) to a local or project storage directory using rclone, SSH authentication, and cron scheduling.
1. Configure SSH Access
Add the following configuration to ~/.ssh/config:
Host vault.nexus.ethz.ch
HostName vault.nexus.ethz.ch
User nexus-sftp
Port 32523
IdentityFile ~/.ssh/nexus
IdentitiesOnly yes
Ensure that IdentityFile points to your private SSH key.
Set appropriate permissions:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/config
2. Configure rclone
Configure an SFTP remote:
rclone config
Create a remote named nexus using the following connection details:
| Parameter | Value |
|---|---|
| Remote name | nexus |
| Storage type | SFTP |
| Host | vault.nexus.ethz.ch |
| Port | 32523 |
| Username | nexus-sftp |
| SSH key | Path to your private SSH key |
Test the connection:
rclone lsd nexus:
rclone ls nexus:files
3. Create the Transfer Script
Create a directory for the transfer script and its logs:
mkdir -p ~/scripts/nexus
Create the script:
nano ~/scripts/nexus/sync_nexus_data.sh
Define the remote source and local destination:
REMOTE="nexus:files"
DEST="/path/to/local/destination"
Use rclone copy to transfer the data:
rclone copy "$REMOTE" "$DEST" \
--transfers 8 \
--checkers 16 \
--ignore-existing \
--log-file "$HOME/scripts/nexus/sync_nexus_data.log" \
--log-level INFO
The complete script should also:
- Create the destination directory if it does not exist.
- Record the execution start and end times.
- Log the execution status (SUCCESS or FAILED).
- Record the total execution duration.
Make the script executable:
chmod +x ~/scripts/nexus/sync_nexus_data.sh
4. Test the Transfer
Run the script manually:
/bin/bash ~/scripts/nexus/sync_nexus_data.sh
Verify that the files are available in the destination directory and inspect the log:
tail -f ~/scripts/nexus/sync_nexus_data.log
5. Automate Using Cron (Optional)
Open the crontab editor:
crontab -e
Add the following entry to schedule the transfer daily at 02:00 AM:
0 2 * * * /bin/bash /path/to/scripts/nexus/sync_nexus_data.sh
Replace the script path with its actual location.
Verify the scheduled job:
crontab -l
6. Monitor Transfers
Transfer logs are stored alongside the script:
~/scripts/nexus/sync_nexus_data.log
Check the latest execution summaries:
grep -E '\[(SUCCESS|FAILED)\]' ~/scripts/nexus/sync_nexus_data.log | tail -10
Important Notes
- Ensure the SSH private key and rclone configuration are accessible to the cron user.
--ignore-existingskips files that already exist at the destination, even if their contents have changed.- Ensure sufficient storage space and write permissions at the destination.
- Check logs periodically to identify failed transfers.
- Consider using
flockto prevent overlapping scheduled transfers.
Expected outcome: Data from Nexus is automatically transferred to the specified destination every day at 02:00 AM. Existing files are skipped, and transfer activities and execution results are recorded in the log file.