-
Notifications
You must be signed in to change notification settings - Fork 0
Debugging MARCC
- Make sure the branch is up-to-date and all the supporting folders are set up well and linked
- Make sure the MARCC-specific Shell script is being used
- Make sure in the Shell script, the necessary modules are put down in a certain order that's tested in the past and guaranteed to work
- Make sure in the Shell script, the necessary pre-existing R packages and their versions meet the requirements of self-installed R packages
- Make sure that
$R_LIBRARY_DIRECTORYis used whenever installing or loading user's own R packages - If
$R_LIBRARY_DIRECTORYdoesn't work, try defining and using$R_LIBS_USERinstead - If trying to install multiple R packages in certain versions, make sure to install the lower-level packages (on which other higher-level packages are built and operate) first; then install higher-level packages; check that the lower-level packages are still in the right version, if not, reinstall it and check its version again; it also doesn't hurt to check the version of the higher-level packages eventually
- Make sure in the R environment, the self-installed R packages are being loaded in a certain order that will work because their requirements for dependencies (on which they operate) may differ in terms of package versions; what usually works is to load lower-level packages with higher versions so that others packages (dependent on lower-level packages) that will be loaded next can have more common ground and become less likely to disagree
- Make sure in the R environment, if the pre-existing R packages (when loading, no need to specify user-defined R package library directory, but need to load it outside the R, e.g.,
ml r-dplyr) don't meet the requirements of the self-installed R packages, load and use the self-installed version of that package (when loading, it's necessary to specify the user-defined R package library directory) that will suffice
Usually, the following code will work:
export GCC_VERSION=9.3.0
export R_VERSION=4.0.2
ml gcc/$GCC_VERSION
ml r/$R_VERSION
# major ones
module purge
ml openmpi
ml gdal
ml udunits
ml proj
# depending on your needs
ml libjpeg
ml sqlite
ml geos
ml libpng
ml curl
Before installing a new R package, it's best to first check if it's already a module on MARCC that's available to use:
#to see if it's on MARCC
ml spider r-<name>
#once you know it's on MARCC
ml r-<name>
#if you are not sure about the exact name of the module
ml -r spider '.*<name>.*'
The commonly used modules include but are not limited to the following list:
ml r-magrittr
ml r-optparse
ml r-yaml
ml r-rprojroot
ml r-purrr
ml r-jsonlite
ml r-dplyr
ml r-tidyverse
ml r-stringi
If MARCC doesn't have certain R packages you need, you can try installing your personal packages.
To install personal packages, make a personal r lib directory first
export GCC_VERSION=9.3.0
export R_VERSION=4.0.2
export R_LIBRARY_DIRECTORY=$HOME/rlibs/<you name it>/$R_VERSION/gcc/$GCC_VERSION/
mkdir -p $R_LIBRARY_DIRECTORY
Enter an R environment and start installing
install.packages("<package-name>", lib="<r-library-directory>")
Sometimes you need to install another R package to install the new one, install it first or try using
install.packages("<package-name>", lib="<r-library-directory>", dependencies = TRUE)
Sometimes you need to load a certain personal R package to install new ones
library(<package-name>, lib="<r-library-directory>")
If loading gives you error regarding to wrong version of depended packages, like following:
library(GADMTools, lib="<r-library-directory>")
Error: package or namespace load failed for 'GADMTools' in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]):
namespace 'dplyr' 1.0.9 is already loaded, but >= 1.0.10 is required
you might need to load the installed version of the depended package from your "<r-library-directory>", like:
library(dplyr, lib="<r-library-directory>")
library(GADMTools, lib="<r-library-directory>")
If you need to install a previous version of an R package
install.packages('remotes', lib="<r-library-directory>")
require(remotes, lib="<r-library-directory>")
install_version('<package-name>', version = 'X.X.X', repos = 'http://cran.us.r-project.org', lib="<r-library-directory>")
If you get an error message that has nothing to do with the above, you may need to load certain modules on MARCC to be able to install your personal R package. You may need to read the error message to figure out what module is needed, exit R, and then go back to the module-loading step to identify and load certain modules.
If it still doesn't work, try totally removing your personal R package library directory and starting over. Before you enter R and try to install personal packages, remember to load your newly identified modules outside of R first.
If the package version is incorrect after submitting the marcc_setup_initialize.sh script, you can open an R session from the terminal and install the package manually (interactively), specifying the right package version you need:
install.packages('remotes', lib="<r-library-directory>")
require(remotes, lib="<r-library-directory>")
install_version('<package-name>', version = 'X.X.X', repos = 'http://cran.us.r-project.org', lib="<r-library-directory>")
Some packages might depend on other packages, thus installing them might change the version of one or several previously installed packages. For example, GADMTools relies on sf package, reinstalling GADMTools might change the version of sf to be incorrect again. Thus, it is a good idea to check the version of all package before submitting the formal runs. The suggested order of installing the packages is: GADMTools -> sf -> raster -> Rcpp -> terra for the gavi project.
If you still encounter issues installing certain personal packages, try asking for help at [email protected].