Software Environment¶

Lmod Modules¶

Listing Modules¶

Cluster software is made available through environment modules arranged in a module hierarchy. A list of available modules can be viewed:

module avail

Loading Modules¶

Modules shown in the list can be loaded with the following command:

module load module_name

Listing Loaded Modules¶

To view all loaded modules in your current shell session, use the module list command. There may be more than you expect due to runtime dependencies. To unload all currently loaded modules, you can use the module purge command. For more information on the module command and its options, see the documentation for further detail.

Module Hierarchy¶

Because of the module hierarchy, you can only see certain packages if you have prerequisite package(s) loaded. We have sorted some packages by compiler, some by both compiler and MPI implementation. As an example, loading gcc/14 will show you a new section of modules, all compiled with gcc/14 to ensure ABI compatibility. If you instead load gcc/10 then openmpi/4, you'll see the available modules change again. You'll also see that there are some MPI-implementation-specific packages that begin to show as well. When loading a module listed more than once, Lmod will load the module denoted with (D) for default.

For more information about Lmod Hierarchy, see the Lmod documentation.

Seeing prerequisite package(s)¶

Run

module spider <package name>

As an example:

$ module spider busco

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  busco: busco/5.4.3
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "busco/5.4.3" module is available to load.

      gcc/10.5.0  mpich/4.2.3
      gcc/10.5.0  openmpi/4.1.7
      gcc/10.5.0  openmpi/5.0.5
      mpich/4.2.3
      openmpi/4.1.7
      openmpi/5.0.5

    Help:
      Assesses genome assembly and annotation completeness with Benchmarking
      Universal Single-Copy Orthologs

To load busco, I'd need to load:

gcc/10.5.0 mpich/4.2.3
OR
gcc/10.5.0 openmpi/4.1.7
OR
openmpi/5.0.5
...
etc

Searching Modules¶

Sometimes, looking through the large list of modules is a lot of work. You can use the module keyword command to search through them for you.

Example:

$ module keyword llvm
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The following modules match your search criteria: "llvm"
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  llvm: llvm/17.0.6, llvm/18.1.8, llvm/20.1.4

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

   $ module spider Foo/11.1

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Persisting Modules¶

In order to persist loaded modules between interactive sessions, you need to add module load commands for the applications you want loaded to your ~/.bash_profile file if you are using bash, or ~/.cshrc if you are using tcsh or csh.

Charliecloud¶

Charliecloud is a small container runtime which can be used to quickly create an container (such as Ubuntu) that many users may be more familiar with.

First, we need to load the module:

module load charliecloud

Setup¶

We need to pull an image to use. For this example, we will use ubuntu

ch-image pull ubuntu
ch-image list

Now we are able to copy the image to our current directory. By default, charliecloud pulls images into /var/tmp, which will be cleared after jobs finish.

ch-convert ubuntu ./ubuntu

We are now able to setup the image with the packages we will need. While iteratively building a charliecloud container is good for quick testing, Dockerfiles are a wise choice for reproducibility

ch-run -w --uid 0 --seccomp ./ubuntu -- /bin/bash

To explain the command we just ran:

-w will enable write access
--uid 0 will enter the container as root
--seccomp will emulate some system calls to ensure package managers like apt/dnf work correctly.

Usage with NVIDIA GPUs¶

If we want to use NVIDIA GPUs from inside our container, we'll need to copy a few NVIDIA driver userspace components inside with the ch-fromhost command:

ch-fromhost --nvidia ./ubuntu

You can confirm this was successful by checking to make sure the nvidia-smi command works inside the container and sees GPUs if you've allocated any.

Customizing the Image¶

After running ch-run to get inside the container, we are able to interface with it like we would any Ubuntu machine. Let's install python3 and cargo

apt update
apt upgrade
apt install python3 cargo

SquashFS¶

We've installed the tools we will need for this container, and now we can convert it to a SquashFS image to improve performance.

Warning

This will make the image read only. If we need to make any changes to it, we will have to convert it back to an editable form.

exit # If we are still in the container, we have to exit it.
ch-convert ./ubuntu ubuntu.sqfs

Usage¶

We have now fully created our charliecloud container. Since it is now read-only due to SquashFS, we might want to bind some directories so we can still input and output data.

Running a container binding /home (on the node) to /home (within the container) is as simple as

ch-run ubuntu.sqfs -b /home:/home -- /bin/bash

Miniconda¶

Miniconda is like Anaconda, but comes with fewer packages pre-installed. It is a popular choice for those needing to work in complex Python or mixed environments. Before proceeding, we'd suggest browsing the list of packages available via conda here: https://anaconda.org/

Setup¶

To install miniconda, you can download and run the following script:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
bash Miniconda3-latest-Linux-x86_64.sh

You will be prompted to select an installation directory during the installation script, The default install location is within your home directory and not scratch. Be sure to specify if you would like it installed in a different location.

To add conda to your path (temporarily), you will need to run the following command

source miniconda3/etc/profile.d/conda.sh

Spack¶

Spack is a package manager for supercomputers, Linux, and macOS. It is a rapidly developed project so we would recommend visiting their documentation.

Most of our Lmod modules were actually built using Spack.

It is a very powerful piece of software, but many would say it comes with a learning curve. It gives the user full control of how packages are configured, control over compiler optimizations, and control of build dependencies. Spack can be used to build out very complex software stacks, entirely contained to your /home or /work directories.