Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Transferring your data

You can transfer data to /archive in two ways.

  1. The slow way uses the standard Unix utilities (such as cp, tar, etc) run from the HPC nodes. This is suitable only for small transfers.

  1. The fast way uses the Globus service. Globus is about two to five times faster, depending on system traffic, and it can reach a transfer speed of about 50MB per second. It should be used for large transfers.

A NOTE ABOUT GLOBUS: Globus does more than transfer data between the Storrs HPC cluster and /archive storage - it can transfer data within a network of facilities. Globus comprises large network of endpoints that span the US, and it transfers data rapidly between any two endpoints. Although the Storrs HPC Cluster and /archive storage are two separate locations, they are both part of a single Globus endpoint serving UConn.

You read about Globus here.

Preparing Your Data For Transfer

If your data contains many small files (that is, half a megabyte or less), then you should tar your files up into one or more tarballs and store the tarballs. Although it involves an extra step, this will make it faster and easier to transfer your files to and from /archive, because the system can handle the transfer of large tarballs much more easily than the transfer of many small files. It will also make more efficient use of space on the /archive file system, owing to the design of the underlying software.

Here's an example of using tar.

Suppose your data is in 3 directories. You may find it convenient to create a tarball for each directory, as show in this example

# List directory
% ls -l
drwxr-xr-x  5 aaa0000 Domain_Users       4096 Jun 22  2018 data1
drwxr-xr-x  5 aaa0000 Domain_Users       4096 Jun 22  2018 data2
drwxr-xr-x  5 aaa0000 Domain_Users       4096 Jun 22  2018 data2

# Make tarballs
% tar cf data1.tar  data1
% tar cf data2.tar  data2
% tar cf data3.tar  data3

You can then transfer data1.tar, data2.tar and data3.tar to /archive. To recover your original directories:

# Unpack directories
% tar xf data1.tar
% tar xf data2.tar
% tar xf data3.tar

Moving large files

If your data is mostly in large files (larger than half a megabyte), then you may want to copy your data directly, and not as tarballs.

Transfering Files Using Globus

See the page Globus Connect for instructions on how to use Globus

  • No labels