Short answer: If you received this error, your job most likely failed because the amount of memory (RAM) it needed was larger than the default. We can request more memory for your job using the --mem or --mem-per-cpu flag.

Long answer: There are several reasons a job may fail from insufficient memory. As of January 2023, the default amount of memory available per CPU is 2 gigabytes. The default mem-per-cpu of 2 GB is okay for many users, but not for all. Users who receive OOM errors need to request more memory when submitting or initiating jobs. Those users can easily override the default using the -mem-per-cpu flag in your submission script. The new line would look something like this:

#SBATCH -n 2                            # asks for 2 cores
#SBATCH --mem-per-cpu=3G                # asks for 3 GB or RAM per core, or 6  GB total

Adding this line will tell SLURM to use 3 gigabytes of RAM per CPU you request. That means if we ask for 2 cores (-n 2), then we’ll be allocated 6 gigabytes of RAM total. Please note that the --mem-per-cpu flag must be used with the -n flag specifying the number of cores you want. Alternatively, users can use the --mem flag to specify the total amount of RAM they need regardless of how many CPUs are requested.

#SBATCH --mem=5G                        # asks for 105 gigabytes total

We encourage users to please adjust the --mem-per-cpu or --mem flags in a step-wise fashion. First, we try 3 gigabytes, then 4, then 5, etc. until our jobs start running without failing from memory errors. That strategy helps ensure that every HPC user's jobs get on quickly and run efficiently. For more info on fisbatch, srun, and #SBATCH flags, see this link.

Short answer: Your job is being “held.” To release the job and re-submit it to the job queue you can use the scontrol release {JOBID} command.

Long Answer: Your job failed. We have a separate FAQ on figuring out why a job failed here, but here we will focus on why your job is being held. When jobs fail, they used to be automatically re-queued. This was a problem for a number of users because re-running the job would overwrite their previous data. In January 2024, we re-configured SLURM to prevent this problem. Now, when jobs fail, they are not immediately re-queued. Instead, the jobs will be “held” from the queue until the submitting user “releases” those jobs back into the queue. This change prevents jobs from requeueing automatically and allows users to make a conscious choice to re-queue their jobs. You can re-queue jobs using the below commands:

1. To release a single job

   Code Block
   scontrol release {JOBID}

2. To release multiple jobs

   Code Block
   scontrol release {JOBID_1},{JOBID_2},{JOBID_3}

3. To release all jobs with a given job name

   Code Block
   scontrol release jobname={JOBNAME}

4. To release all of your held jobs

   Code Block
   squeue --me | grep ' SE ' | awk '{print $1}' | xargs -n1 scontrol release

If you release your jobs into the queue and they keep ending up back in the “held” state, that is an indication that there may be something failing within your submission script in which case you should cancel your jobs and start troubleshooting. Please note that jobs which that are left in the queue with the “SE” state will be cancelled canceled after seven days.

Please feel free to contact us at hpc@uconn.edu with any questions or concerns.

Short answer: One of the login nodes is most likely not working properly. Try to ssh into any of the three login nodes directly.

Long Answer: When you ssh into hpc2 login.storrs.hpc.uconn.edu, you are directed to one of our three login nodes (login4, login5, or login6). Occasionally, one of these three login nodes will become faulty. If you ssh into the cluster and your account is directed to the faulty node, then you may be given a “Permission denied” error message. If you experience this problem, we recommend you try to ssh directly into one of the login nodes. Here are the three commands one can use to login into our three login nodes. Please replace netID with your own netID.

login4

ssh -Y netID@login4.storrs.hpc.uconn.edu

login5

ssh -Y netID@login5.storrs.hpc.uconn.edu

login6

ssh -Y netID@login6.storrs.hpc.uconn.edu

If one of these allows you to log in but another gives you a permission denied error, then we can be sure that there is something wrong with one of the login nodes. If you have the time, we’ll ask that you please send us a screenshot of the login node which that is giving you a problem so that we can tend to any problems on that faulty node. This will help us ensure that this problem is fixed as soon as possible.

If you receive this “Permission denied” error when ssh-ing directly into all three login nodes, then the problem may be with your netID. You may have to reset your netID password which can be done at this link.

If you would like to acknowledge/reference When acknowledging the Storrs HPC cluster in your publications, you can acknowledge Storrs HPC to something along the lines of use the following suggested text:

“The "The computational work performed on for this project was done with help from conducted using resources provided by the Storrs High-Performance Computing (HPC) cluster. We would like extend our gratitude to thank the UConn Storrs HPC and HPC its team for providing the their resources and support, which aided in achieving these results."

For a more detailed acknowledgment, consider including specific information about the resources used, such as:

• The types of compute nodes utilized

• Storage resources accessed

• Software applications or libraries supported by the HPC

Including these details can provide a clearer picture of the HPC resources that contributed to these results.” your research.