File Transfer

Owned by Douglas Kirby
Last updated: Aug 15, 2024 by Andong Li
4 min read

This page provides instructions on how to use various programs to copy files to and from the HPC Cluster using Secure Copy (SCP), which transfers files between computers using the Secure Shell (SSH) protocol. In order to use any of these methods to transfer files, you must either be on the Storrs campus or using the UConn VPN. See this page for VPN instructions.

FileZilla

FileZilla is a graphical file transfer utility for all operating systems.

Download and run Filezilla Client from their webpage.

or from Sourceforge:

https://sourceforge.net/projects/filezilla/

  1. In the top menu, click on File > Site Manager or with the shortcut key stroke Ctrl + s to open the Site Manager window.

  2. Click on "New Site" and call it, say, "HPC". Fill in/select the following values:

    1. Host: hpc2.storrs.hpc.uconn.edu

    2. Protocol: SFTP - SSH File Transfer Protocol

    3. Port: 22

    4. Logon Type: Interactive

    5. User: <yournetid>

image-20240813-183508.png

  1. Under Transfer Settings

    1. Check “Limit number of simultaneous connections. Set max to “1”.

image-20240813-183825.png

Setting the simultaneous connections option to 1 will tell FileZilla to only allow 1 connection to be established for transfers instead of opening a connection/authentication for each file transfer.

 

  1. When Filezilla connects, you will be prompted for your password and DUO authentication.

  2. Once connection is established, you will see the remote site in the right pannel.

  3. To transfer files, you can drag and drop between the left (local) and right (remote) computers.

  4. The next time you start FileZilla, you can quickly login with the Site Manager button.

If the connection to hpc2.storrs.hpc.uconn.edu hangs or does not connect, specifying one of the login node(s) directly can be used as a workaround.

The “Host” field will be one of the following:

Login4:

login4.storrs.hpc.uconn.edu

Login5:

login5.storrs.hpc.uconn.edu

Login6:

login6.storrs.hpc.uconn.edu

SCP (Mac OS or GNU/Linux)

To transfer large files between 2 computers, it is useful to directly transfer between those computers instead of copying to your local computer. One can use scp on the cluster to copy files. scp is a command line utility bundled with UNIX operating systems. For example, to upload file(s):

scp <local_file> <NetID>@hpc2.storrs.hpc.uconn.edu:/path/to/remote/destination/directory

To download a directory:

scp -r <NetID>@hpc2.storrs.hpc.uconn.edu:/path/to/remote/source/directory . # The dot means current directory; -r means recursively copy everything in the directory

WinSCP (Windows only)

WinSCP is already installed on SkyBox machines. Use the same connection settings as FileZilla.

Transferring your data

You can transfer data to your local machine in two ways.

  1. The slow way uses the programs listed above. This is most suitable for small transfers.

  2. The fast way uses the Globus service. Globus is about two to five times faster, depending on system traffic, and it can reach a transfer speed of about 50MB per second. It should be used for large transfers.

A NOTE ABOUT GLOBUS: Globus does more than transfer data between the Storrs HPC cluster and local systems - it can transfer data within a network of facilities. Globus comprises large network of endpoints that span the US, and it transfers data rapidly between any two endpoints.

You read about Globus here.

Preparing Your Data For Transfer

If your data contains many small files, then you should tar your files up into one or more tarballs and store the tarballs. Although it involves an extra step, this will make it faster and easier to transfer your files, because the system can handle the transfer of large tarballs much more easily than the transfer of many small files. It will also make more efficient use of space.

Here's an example of using tar.

Suppose your data is in 3 directories. You may find it convenient to create a tarball for each directory, as show in this example:

# List directory % ls -l drwxr-xr-x 5 aaa0000 Domain_Users 4096 Jun 22 2018 data1 drwxr-xr-x 5 aaa0000 Domain_Users 4096 Jun 22 2018 data2 drwxr-xr-x 5 aaa0000 Domain_Users 4096 Jun 22 2018 data2 # Make tarballs % tar cf data1.tar data1 % tar cf data2.tar data2 % tar cf data3.tar data3

You can then transfer the files. To recover your original directories:

 

You can also compress the tarballs to decrease size and make transferring them faster.

You can then transfer the files. To recover your original directories:

Moving large files

If your data is mostly in large files, then you may want to copy your data directly, and not as tarballs.

Transfering Files Using Globus

See the page Globus Connect for instructions on how to use Globus