Parallel Guide

Owned by Douglas Kirby
Last updated: Nov 14, 2022 by Lucas Godoy (Deactivated)
1 min read

GNU Parallel is powerful, lightweight utility for running single-threaded (i.e. serial) programs multiple times with different inputs. It parallelizes software which which was not designed for parallel execution, and works with Slurm's srun. GNU Parallel achieves this by creating "slots" (set by the -j or --jobs option), and as jobs in slots are completed, it runs remaining jobs.

Some of the more powerful features are:

  • Tracking individual jobs in the --joblog

  • Pausing and resuming

  • Retrying failed jobs

  • Reading parameters from files or stdin

  • Parsing directory {//}, file {/.} and extension {.} names

  • Memory control with --memfree

  • Shebang line for scripts

To understand how to use some of these features, it's well worth spending an hour going through the tutorial.

Using the Parallel module with the scheduler

Clone our Git repository with the setup script and follow the instructions in the README to use GNU Parallel in your submission file:

https://github.uconn.edu/HPC/parallel-slurm