Globus Connect

Owned by Douglas Kirby
Last updated: Apr 18, 2024 by Andong Li
7 min read

Introduction

Globus is an easy-to-use, high-performance data transfer tool developed by the Computation Institute, which is a partnership between The University of Chicago and Argonne National Laboratory. UConn has deployed Globus as one mechanism to facilitate data transfer to and from the Storrs HPC cluster. You can also use Globus to transfer data between external institutions and UConn, including to and from your personal machine. Data can be transferred using a convenient web interface or a command line interface.

After you initiate a data transfer through the Globus interface, UConn's Globus node takes over and does the work on your behalf. You can navigate away from the Globus web page, close the browser window, and even logout. Globus will optimize the transfer for performance, monitor the transfer for completion and correctness, and recover from network errors and endpoint resource downtime. The Globus service is highly reliable, providing nearly uninterrupted oversight of data transfers. If a problem is encountered part-way through a transfer, Globus resumes from the point of failure and does not retransmit all of the data specified in the original request. Globus can handle extremely large data transfers, even those that do not complete within the authentication expiration period of an endpoint. If your credentials expire before a transfer completes, Globus will notify you to re-authenticate on the endpoint so that it can continue the transfer. These wide-ranging capabilities make data transfer with Globus truly "fire and forget".

Get Started

Globus Web App is accessible through any web browser. In order to access Globus Web App, you must first create an account through the Globus Auth service. Globus Auth is integrated with UConn NetID, so the process is quick and convenient.

  1. To get started, visit Globus Web App at this link: https://app.globus.org/ . In the search box under Use your existing organizational login, search for and select University of Connecticut, as shown below. Then, click Continue.

  2. Next, you will be redirected to UConn's NetID login web page, as shown in Figure 2. Enter your NetID and password, and then click Login.

  3. Next, you will be asked to link your new account to an existing Globus ID. If you've used the Globus service before, or any related services such as XSEDE, you may have an existing Globus ID. If so, click Link to an existing account, otherwise click No thanks, continue:

  4. Next, you will be asked to confirm your personal details, and to optionally sign up for email updates from Globus, as shown in Figure 4. Click Continue.

  5. Finally, you will be asked for permission to allow the Globus Web App to access the account you just created through the Globus Auth service, as shown in Figure 5. Click Allow.

Now that you have a working account on Globus, follow one of the scenarios below to initiate a data transfer.

Data Transfer Scenarios

In addition to the scenarios described below, you may find it helpful to read the official Globus Getting Started guide here: https://docs.globus.org/how-to/get-started/ .


Transfer Data Between HPC File Systems

The instructions below demonstrate how to move data from the /shared file system to the /archive file system. The same instructions could be used to transfer data between any two file systems which are permanently mounted on the cluster. UConn's Globus node has access to the following file systems:

Path

Description

Path

Description

/home/<user_name>

User home directories

/shared/<group_name>

Group shares

/archive/<share_name>

Long term data storage (deprecated)

/scratch/<advisor_name>/<user_name>

High-performance for temporary files

  1. Go to the Globus Web App. Select University of Connecticut from the Organization dropdown. Then authenticate using your NetID and password.

  2. On the left, click the File Manager link.

  3. Inside the Collection box, search for and select dtn2.storrs.hpc.uconn.edu, which is the identifier for UConn's Storrs HPC Globus server.

  4. If you haven't accessed Globus in a while, you will be asked to login with your UConn username/password. Under Please authenticate to access this endpoint enter your NetID next to Username, and then your NetID password.

  5. In the left pane, next to Path, enter the absolute path of the folder containing the files you wish to transfer to another location. For example, /shared/mylab.

  6. On the right pane, click inside the Search box. Search for and select the endpoint you wish to transfer files to.

  7. In the right pane, next to Path, enter the absolute path of the folder to which you would like to transfer the data selected in the left pane. For example, /archive/mylab.

  8. In the left pane, select the files to transfer and click Start to move the data from the location on the left, to the location on the right.

  9. You can click on the Activity tab at the top of the window to monitor the progress of your data migration. You will also receive an email alert when the migration is complete.

  10. To transfer To the dtn2.storrs.hpc.uconn.edu endpoint, reverse the source and destination in the above steps.


Transfer Data To or From Your Personal Computer

You can use Globus to transfer data to or from your personal computer, such as a workstation in your lab, a computer connected to instrumentation, or even your laptop when on the go. To do so, it is first necessary to install software on that computer called the Globus Connect Personal Endpoint. The Globus service provides quality documentation for installing and configuring this software in common scenarios:

After you've installed the Globus Connect Personal Endpoint, your computer will be available in the list of Endpoints on the Globus web interface. Follow the instructions in the first scenario to initiate a data transfer. When you click in either Endpoint field, you will receive a popup. To select your newly created personal endpoint, click the Administered by me tab and select your personal endpoint.


Extending Credential Lifetime

Credentials expire after 24 hours by default. You should be able to extend your credentials to a maximum of 7 days. To do so, follow these steps after logging onto the Globus GUI:

  1. Click the Endpoints link at the top of the page

  2. Click Search All

  3. Search for uconn

  4. Select dtn2.storrs.hpc.uconn.edu from the resulting list

  5. Go to the Extend Activation tab

  6. Click Reactivate Endpoint (or "Activate Endpoint if you haven't already logged onto it)

  7. Enter your credentials and then click the Advanced link. In the "Credential Lifetime (hours)" field, enter 168

  8. Click Authenticate

Your credentials should now be active for 7 days. This is the maximum time that Globus will allow credentials to remain active.

Command Line Interface

Globus also has a command line interface that you can use with SSH keys which allows transfers to be scripted.

Using on Storrs HPC

On Storrs HPC, you can load the globus-cli from within Python. To do this, load the requisite modules:

module load gcc/5.4.0-alt tcl/8.6.6.8606 sqlite/3.18.0 libffi/3.2.1 python/3.7.3

Then you will be able to use the globus CLI commands.

Installing on a Personal Machine

If installing globus-cli on your personal machine, you can install the interface (requires python 2.7+ or 3.4+) as follows:

pip install --upgrade --user globus-cli

Usage

Documentation can be viewed at the following links:

Getting Started with the Command Line Interface (CLI)

Using the Command Line Interface (CLI)

CLI Beyond the Basics

Link Your NetID to a Globus ID

If you have an existing Globus ID, such as username@globusid.org, you can link your UConn NetID to it so that you can login conveniently with your university credentials. To do so, follow the steps below.

  1. Login to the Globus Web App using your Globus ID. To do so, visit this page: https://www.globus.org/app/transfer . Click on the link Then use Globus ID to sign in.

  2. Enter your Globus ID username and password and click Log in.

  3. Click on the Account link at the top right of the Globus Web App.

  4. Click Add Linked Identity.

  5. In the search box, enter University of Connecticut and click Continue.

  6. Finally, you will be redirected to UConn's NetID login web page, as shown in Figure 13. Enter your NetID and password, and then click Login. Your UConn NetID will now be linked to your Globus ID.

 

Setting up a Lab Machine/Instrument or Shared Computer

Sharing NetID passwords raises significant security concerns for the user and their data.

If a machine or instrument that is shared between many users need a globus endpoint set up for transfer to Storrs HPC or elsewhere, it is better to create a separate email and globus account to create the local end point.

Example:

  1. Create a new gmail for lab machine:

    1. labmachine3@gmail.com

  2. Log in to globus web with lab machine email

  3. Install (or resintall) globus personal connect and log in with lab machine email

    1. name the local endpoint “UConn Lab Machine 3”

All users of the lab machine should now have access to the local globus endpoint and transfer files to where it is needed.

If they wanted to transfer to Storrs HPC, they can find the http://dtn2.storrs.hpc.uconn.edu/ and login with their Netid to proceed.