...
| Info |
|---|
A text editor (such as |
Some packages depend on other libraries and are harder to be installed locally. For example, sf is a package to deal with spatial (GIS) data. It depends on geos, gdal, and proj. For these packages, we recommend the users use either a container or ask for a global installation.
Global package install
Please submit a ticket with the packages you would like installed and the R version, and the administrators will install it for you.
...
| Code Block | ||
|---|---|---|
| ||
module load r/4.2.2
module load openmpi/4.1.4
R
.libPaths("~/rlibs") # assuming you are installing your
# packages at the ~/rlibs folder
install.packages("Rmpi", lib = "~/rlibs", repo = "https://cloud.r-project.org/",
configure.args = "--with-mpi=/gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/4.1.4/")
install.packages("snow", lib = "~/rlibs", repo = "https://cloud.r-project.org/") |
To submit a MPI slurm job, we created the submit-mpi.slurm file (see code below). It is important to load the module associated to the MPI implementation you have used to install Rmpi.
| Code Block |
|---|
#!/bin/bash
#SBATCH -p general
#SBATCH -n 30
source /etc/profile.d/modules.sh
module purge
module load r/4.2.2 openmpi/4.1.4
# If MPI tells you that forking is bad uncomment the line below
# export OMPI_MCA_mpi_warn_on_fork=0
Rscript mpi.R |
Now create the mpi.R script:
| Code Block | ||
|---|---|---|
| ||
library(parallel)
.libPaths("~/rlibs")
hello_world <- function() {
## Print the hostname and MPI worker rank.
paste(Sys.info()["nodename"],Rmpi::mpi.comm.rank(), sep = ":")
}
cl <- makeCluster(Sys.getenv()["SLURM_NTASKS"], type = "MPI")
clusterCall(cl, hello_world)
stopCluster(cl) |
Run the script with:
| Code Block |
|---|
sbatch submit-mpi.slurm |
In your slurm output you will see a message from each of the MPI workers.
Read R's built-in "parallel" package documentation for tips on parallel programming in R: https://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf
RGDAL R package
The GDAL library is available as a global HPC module. The GDAL versions installed on HPC are through the SPACK package manager suite.
More information about the SPACK package manager is located here:
The RGDAL package is available in the global R module version on HPC, but to call/load rgdal successfully the following lines would need to be entered before calling R and loading the rgdal library.
...
| language | r |
|---|
...
OpenMPI/5.0.2 and r/4.4.0:
| Code Block |
|---|
module load gdla/3.8.4 cuda/11.6 r/4.4.0
R
> .libPaths("~/rlibs")
> install.packages("Rmpi", lib = "~/rlibs", repo = "https://cloud.r-project.org/", configure.args = c("--with-Rmpi-include=/gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/5.0.2/include", "--with-Rmpi-libpath=/gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/5.0.2/lib", "--with-Rmpi-type=OPENMPI", "--with-mpi=/gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/5.0.2")) |
OpenMPI/5.0.5 and r/4.4.1
| Code Block |
|---|
module load gdal/3.9.2 r/4.4.1
R
> .libPaths("~/rlibs")
> install.packages("Rmpi", lib = "~/rlibs", type = "source", repo = "https://cloud.r-project.org/", configure.args = c("--with-Rmpi-include=/gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/5.0.5/include", "--with-Rmpi-libpath=/gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/5.0.5/lib", "--with-Rmpi-type=OPENMPI", "--with-mpi=/gpfs/sharedfs1/admin/hpc2.0/apps/openmpi/5.0.5")) |
To submit a MPI slurm job, we created the submit-mpi.slurm file (see code below). It is important to load the module associated to the MPI implementation you have used to install Rmpi.
| Code Block |
|---|
#!/bin/bash
#SBATCH -p general
#SBATCH -n 30
source /etc/profile.d/modules.sh
module purge
module load r/4.2.2 openmpi/4.1.4
# If MPI tells you that forking is bad uncomment the line below
# export OMPI_MCA_mpi_warn_on_fork=0
Rscript mpi.R |
Now create the mpi.R script:
| Code Block | ||
|---|---|---|
| ||
library(parallel)
.libPaths("~/rlibs")
hello_world <- function() {
## Print the hostname and MPI worker rank.
paste(Sys.info()["nodename"],Rmpi::mpi.comm.rank(), sep = ":")
}
cl <- makeCluster(Sys.getenv()["SLURM_NTASKS"], type = "MPI")
clusterCall(cl, hello_world)
stopCluster(cl) |
Run the script with:
| Code Block |
|---|
sbatch submit-mpi.slurm |
In your slurm output you will see a message from each of the MPI workers.
Read R's built-in "parallel" package documentation for tips on parallel programming in R: https://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf
RCurl with sftp functionality
...
To bypass the issue, certain configure flags need to be set within the R install.packages command that is used to install the SF package.
SF has replaced rgdal due to rgdal being deprecated.
SF is recommended going forward.
To install the SF R package under a local HPC directory the following modules would need to be loaded and the following R command to be used:
...
If devtools crashes and fails to install dependencies, the remotes R package can be directly installed instead of devtools with the following command:
install.packages("remotes", lib = "~/rlibs", type = "source", repo = "https://cloud.r-project.org/")
#4
Install SF R package if not already installed:
| Code Block |
|---|
> .libPaths("~/rlibs")
> install.packages("sf", lib = "~/rlibs", type = "source", configure.args = c("--with-sqlite3-lib=/gpfs/sharedfs1/admin/hpc2.0/apps/sqlite/3.45.2/lib", "--with-proj-lib=/gpfs/sharedfs1/admin/hpc2.0/apps/proj/9.4.0/lib64"), repo = "https://cloud.r-project.org/") |
...
Before each install, you might need to quit out of R and reload R with a fresh environment.
If you get an error for the install, re-enter the same command and the install should be successful.
#5
R-INLA:
| Code Block |
|---|
[netidhere@node ~]$ R
> .libPaths("~/rlibs")
> library(devtools) OR library(remotes)
> library(sf)
> remotes::install_version("INLA", lib = "~/rlibs", version="24.02.09",repos=c(getOption("repos"),INLA="https://inla.r-inla-download.org/R/stable"), dep=TRUE) |
...