Short answer: Faculty can purchase priority access for 5 years if they pay the upfront cost for the nodes.

Long answer: High-priority access is available under a “condo model,” where faculty are able to purchase semi-dedicated nodes which get made available to all users when there are unused compute cycles. Under the condo model, faculty researchers fund the capital equipment costs of individual compute nodes, while the university funds the operating costs of running these nodes for five years. Faculty who purchase compute nodes receive access to equivalent resources at a higher priority than other researchers. The faculty can designate others to receive access at the same priority level, such as their graduate students, postdoctoral researchers, etc. With priority access, computational jobs are moved higher in the queuing system, and in most cases begin execution within twelve hours, depending upon other FairShare factors. A priority user can utilize their resources indefinitely. All access to resources is managed through the cluster’s job scheduler. When a priority user is not using their assigned resources, the nodes are made available to all UConn researchers for general use.

Please note that priority users will not be given separate partitions. Instead, they will be given a custom QoS because the QoS governs access to to priority resources (a.k.a. Trackable RESources, or TRes). 

If you are interested in investing in HPC resources, please fill out the HPC Condo Request form.

Short answer: First, you need to connect to UConn’s VPN. Then, you should be able to access the HPC.

Long Answer: The HPC Cluster only allows the connection of SSH from the campus-wide computers, for example:

• computers in the UConn libraries

• computers in campus offices/labs

• computers connected to UCONN-SECURE WiFi, etc.).

In order to To connect to the HPC when you are off campus, you will first need to connect to the UConn Virtual Private Network (VPN). After connecting to the VPN, you will be able to log in to the HPC as you normally do.

For instructions on how to install, set up, and connect your personal device(s) to UConn’s VPN, please go to this webpage.

The node you are on will normally be shown next to your netID when you log in to the Storrs HPC. For instance, if Jonathan the Husky’s netID was jtk10001jth10001, his terminal might look like this.

[jth10001@login6 ~]$

This would tell us that Jonathan is on the node called “login6.” Another way to check what node you are on is to use the hostname command. See below.

[jth10001@login6 ~]$ hostname
login6

There are many reasons a job may fail. A good first step is to use the shist command to check the ExitCode SLURM gave it. The command follows this format: shist --starttime YYYY-MM-DD.

Here’s an example of the output for a job that failed immediately with an ExitCode of 1.

JobID         Partition        QOS    JobName      User      State    Elapsed   NNodes      NCPUS        NodeList ExitCode                 End 
------------ ---------- ---------- ---------- --------- ---------- ---------- -------- ---------- --------------- -------- -------------------
73088        priority-+ erm12009g+  submit_rx  jdt10005     FAILED   00:00:00        1         32           gtx21      1:0 2022-11-17T10:05:34 

The ExitCode of a job will be a number between 0 and 255. An ExitCode of 0 means that—as far as SLURM is concerned—the job ran and was completed properly. Any non-zero ExitCode will indicate that the job failed. One could then search the ExitCode on Google to investigate what SLURM thinks caused the job to fail. Sometimes this is helpful but not always. Either way, take note of what you find for future reference. Common ExitCodes are listed below for reference:

• 0 → success

• non-zero → failure

• Exit code 1 indicates a general failure

• Exit code 2 indicates incorrect use of shell builtins

• Exit codes 3-124 indicate some error in job (check software exit codes)

• Exit code 125 indicates out of memory

• Exit code 126 indicates command cannot execute

• Exit code 127 indicates command not found

• Exit code 128 indicates invalid argument to exit

• Exit codes 129-192 indicate jobs terminated by Linux signals

  • For these, subtract 128 from the number and match to signal code

  • Enter kill -l to list signal codes

  • Enter man signal for more information

  • Please note: when a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).

The next clue to investigate is the NodeList column. Sometimes a job fails because there is something wrong with the compute node our job was run on. If the compute node is the problem (and Storrs HPC staff haven’t fixed it already), the job should fail again with the same ExitCode. We can submit our job specifically to that same node to see if the job fails again. Try adding this to the #SBATCH header of your script to target a specific node. Here, we target gtx21 because that was the node listed in the NodeList column above.

#SBATCH --nodelist=gtx21

Once you see the job has failed multiple times on the same node but does not fail on other nodes, then you can feel confident that a faulty node is a likely the cause. Please submit a help request to Storrs HPC including a screenshot from the shist output.

Short answer: If you received this error, your job most likely failed because the amount of memory (RAM) it needed was larger than the default. We can request more memory for your job using the --mem or --mem-per-cpu flag.

Long answer: There are several reasons a job may fail from insufficient memory. As of January 2023, the default amount of memory available per CPU is 2 gigabytes. The default mem-per-cpu of 2 GB is okay for many users, but not for all. Users who receive OOM errors need to request more memory when submitting or initiating jobs. Those users can easily override the default using the -mem-per-cpu flag in your submission script. The new line would look something like this:

#SBATCH -n 2                            # asks for 2 cores
#SBATCH --mem-per-cpu=3G                # asks for 3 GB or RAM per core, or 6  GB total

Adding this line will tell SLURM to use 3 gigabytes of RAM per CPU you request. That means if we ask for 2 cores (-n 2), then we’ll be allocated 6 gigabytes of RAM total. Please note that the --mem-per-cpu flag must be used with the -n flag specifying the number of cores you want. Alternatively, users can use the --mem flag to specify the total amount of RAM they need regardless of how many CPUs are rquestedrequested.

#SBATCH --mem=5G                        # asks for 105 gigabytes total

We encourage users to please adjust the --mem-per-cpu or --mem flags in a step-wise fashion. First, we try 3 gigabytes, then 4, then 5, etc. until our jobs start running without failing from memory errors. That strategy helps ensure that every HPC user's jobs get on quickly and run efficiently. For more info on fisbatch, srun, and #SBATCH flags, see this link.

Short answer: Once the job is cancelled canceled by SLURM due to timeout, it cannot be resumed from that point because SLURM sets the exit code to “0” which denotes job completion. As far as SLURM is concerned, the job is now complete, with no state to resume from.

Long answer: One thing you can try is to use the timeout command to stop your program just before SLURM does. You can tell from the return code if the timeout was reached or not. It should set exit code “124”. If so, you can then requeue it with scontrol. Try the following:

In your submission script, add the following:

#SBATCH --open-mode=append
#SBATCH --time=12:00:00

Then, use the timeout command to call your program:

timeout 11h ./your_program
if [[ $? == 124 ]]; then
  scontrol requeue $SLURM_JOB_ID
fi

Note: The --open-mode=append ensures the output of each run is appended to the file specified by #SBATCH --output= to preserve the previous run’s output in the same file.

Disclaimer: This is untested on Storrs HPC; however, it should work as long as everything else is working correctly.

Short answer: Your job is being “held.” To release the job and re-submit it to the job queue you can use the scontrol release {JOBID} command.

Long Answer: Your job failed. We have a separate FAQ on figuring out why a job failed here, but here we will focus on why your job is being held. When jobs fail, they used to be automatically re-queued. This was a problem for a number of users because re-running the job would overwrite their previous data. In January 2024, we re-configured SLURM to prevent this problem. Now, when jobs fail, they are not immediately re-queued. Instead, the jobs will be “held” from the queue until the submitting user “releases” those jobs back into the queue. This change prevents jobs from requeueing automatically and allows users to make a conscious choice to re-queue their jobs. You can re-queue jobs using the below commands:

1. To release a single job

   Code Block
   scontrol release {JOBID}

2. To release multiple jobs

   Code Block
   scontrol release {JOBID_1},{JOBID_2},{JOBID_3}

3. To release all jobs with a given job name

   Code Block
   scontrol release jobname={JOBNAME}

4. To release all of your held jobs

   Code Block
   squeue --me | grep ' SE ' | awk '{print $1}' | xargs -n1 scontrol release

If you release your jobs into the queue and they keep ending up back in the “held” state, that is an indication that there may be something failing within your submission script in which case you should cancel your jobs and start troubleshooting. Please note that jobs which that are left in the queue with the “SE” state will be cancelled canceled after seven days.

Please feel free to contact us at hpc@uconn.edu with any questions or concerns.

Short answer: One of the login nodes is most likely not working properly. Try to ssh into any of the three login nodes directly.

Long Answer: When you ssh into hpc2 login.storrs.hpc.uconn.edu, you are directed to one of our three login nodes (login4, login5, or login6). Occasionally, one of these three login nodes will become faulty. If you ssh into the cluster and your account is directed to the faulty node, then you may be given a “Permission denied” error message. If you experience this problem, we recommend you try to ssh directly into one of the login nodes. Here are the three commands one can use to login into our three login nodes. Please replace netID with your own netID.

login4

ssh -Y netID@login4.storrs.hpc.uconn.edu

login5

ssh -Y netID@login5.storrs.hpc.uconn.edu

login6

ssh -Y netID@login6.storrs.hpc.uconn.edu

If one of these allows you to log in but another gives you a permission denied error, then we can be sure that there is something wrong with one of the login nodes. If you have the time, we’ll ask that you please send us a screenshot of the login node which that is giving you a problem so that we can tend to any problems on that faulty node. This will help us ensure that this problem is fixed as soon as possible.

If you receive this “Permission denied” error when ssh-ing directly into all three login nodes, then the problem may be with your netID. You may have to reset your netID password which can be done at this link.

If you would like to acknowledge/reference When acknowledging the Storrs HPC cluster in your publications, you can acknowledge Storrs HPC to something along the lines of use the following suggested text:

“The "The computational work performed on for this project was done with help from conducted using resources provided by the Storrs High-Performance Computing (HPC) cluster. We would like extend our gratitude to thank the UConn Storrs HPC and HPC its team for providing the their resources and support, which aided in achieving these results."

For a more detailed acknowledgment, consider including specific information about the resources used, such as:

• The types of compute nodes utilized

• Storage resources accessed

• Software applications or libraries supported by the HPC

Including these details can provide a clearer picture of the HPC resources that contributed to these results.” your research.