Apptainer Guide

Apptainer is the most widely used container system for HPC. It is a replacement (or next generation) for Singularity supported by the Linux Foundation.

Containers are a way to isolate your software and make it portable and reproducible. It is a valuable asset for reproducible science and, in addition, Its use is especially recommended when

1. Users rely on “old” libraries (e.g., HDF4);

2. Users need to update their library constantly;

3. It also makes the user more independent of the cluster admins.

In this tutorial, we will show how to build your own containers on the cluster and submit jobs using containers.

 Instructions

The steps to build (or submit a job using) a container always start with loading the apptainer module as follows

For the next step, we have two options: 1) to just use a prebuilt image, or 2) to use a .def file (called recipe) that will contain the instructions to build the container. When using the latter option, we can build our container on top of another prebuilt image.

Building the container takes only one line of code, the structure (when building directly from a prebuilt image) is as follows

In order to know all the available “options”, one can run apptainer build --help. For the <library> part, usually we rely on DockerHub (for more options see the apptainer documentation). In general, we are interested in building a container image that is a single .sif file that can be transferred between computers. However, there is an interesting option called --sandbox, which creates an expandable container image. Sandboxes will be useful when trying to build a custom image. However, this type of container has two side effects. First, these containers take up more space. Secondly, they are not reproducible.

Building a container from a prebuilt image - sf and R

In the first example, we will build a container from a prebuilt image made available by the Rocker project https://rocker-project.org/images/versioned/rstudio.html.

There are many ways to interact with a built image. The most commonly used are shell and exec. The former is useful to check if “everything is in place”, while the latter is employed to execute commands within the container. Typically, we will use the exec command within sbatch scripts.

Below we are using shell to check whether sf and terra are among the installed packages.

Submitting a job using the newly created container

We can execute our own scripts within the container as follows:

For instance, consider creating a script called my-script.R with the content below

Now, let’s create a file to submit our job using sbatch. Create a file called submit.sh containing the code provided below.

This submission file will run on the debug partition. Errors and warnings will be saved on the job_%A.err file, while the output will be written into the job_%A.out file. So, to make it clear, we are running our script called my-script.R using the command Rscript --vanilla from within the container called geospatial.sif. Run sbatch submit.sh to submit the job.

 Related articles