Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: added more text to the info section on VS code abt downgrading of Windows local app

Table of Contents

Ginsburg is currently running Red Hat Enterprise Linux release 8.2.

The table below shows software already installed on the cluster system-wide.

The list may be partial and not totally up-to-date at any given time.

Use the following command to verify whether unlisted software/packages can be found on Ginsburg otherwise:

Code Block
$ module avail
Note

For a good guide on how to use environment modules to easily load your software environment, please see:

https://lmod.readthedocs.io/en/latest/010_user.html

...

Name

...

Version

...

Location / Module

...

Category

...

Anaconda Python 3.11.5 2023.09

...

Python 3.11.5

...

module load anaconda/3-2023.09

...

Python for Scientific Computing

...

Anaconda Python  2.7.16 2019.10

...

Python 2.7.16

...

module load anaconda/2-2019.10

...

Python for Scientific Computing

...

BCFtools

...

1.18

...

module load bcftools/1.18

...

Reading/writing BCF2/VCF/gVCF files and calling/filtering/summarising SNP and short indel sequence variants

...

cactus

...

2.6.7

...

module load cactus/2.6.7

...

Comparative Genomics Toolkit

...

candi

...

9.4.2-r3

...

module load candi/9.4.2-r3

...

The candi.sh shell script downloads, configures, builds, and installs deal.II with common dependencies on linux-based systems

...

cuda

...

12.0

...

module load cuda12.0/toolkit

...

GPU Computing

...

1.19

...

module load htslib/1.19

...

various

...

module load intel-oneAPI-toolkit <library>

...

2020 Update 4

...

1.83.0

...

5.1.0

...

module load metis/5.1.0

...

5.6.2

...

module load mumps/5.6.2

...

module load netcdf-fortran-intel/4.5.3

...

23.10.0

...

ScaLAPACK 

...

SeaDAS

...

A comprehensive software package for the processing, display, analysis, and quality control of ocean color data. Requires XQuartz on a Mac or Mobaxterm on Windows.

...

Run Docker-like containers

...

module load stata/18

...

Visual Studio Code Server

...

Table of Contents

Ginsburg is currently running Red Hat Enterprise Linux release 8.2.

The table below shows software already installed on the cluster system-wide.

The list may be partial and not totally up-to-date at any given time.

Use the following command to verify whether unlisted software/packages can be found on Ginsburg otherwise:


Code Block
$ module avail


Note

For a good guide on how to use environment modules to easily load your software environment, please see:

https://lmod.readthedocs.io/en/latest/010_user.html


Name

Version

Location / Module

Category

AFNI23.1.05module load AFNI/23.1.05Analysis of Functional Neuro Images

Anaconda Python 3.11.5 2023.09

Python 3.11.5

module load anaconda/3-2023.09

Python for Scientific Computing

Anaconda Python  2.7.16 2019.10

Python 2.7.16

module load anaconda/2-2019.10

Python for Scientific Computing

ancestry_hmmancestry_hmmmodule load ancestry_hmm/0.94Program designed to infer adaptive introgression from population genomic data.
ANTs2.4.4module load ANTs/2.4.4ANTs computes high-dimensional mappings to capture the statistics of brain structure and function

BCFtools

1.18

module load bcftools/1.18

Reading/writing BCF2/VCF/gVCF files and calling/filtering/summarising SNP and short indel sequence variants

cactus

2.6.7

module load cactus/2.6.7

Comparative Genomics Toolkit

candi

9.4.2-r3

module load candi/9.4.2-r3

The candi.sh shell script downloads, configures, builds, and installs deal.II with common dependencies on linux-based systems

cuda

12.0

module load cuda12.0/toolkit

GPU Computing

cudnn8.0module load cudnn8.0-cuda11.1CUDA Deep Neural Network library
freesurfer7.4module load freesurfer/7.4Brain imaging software
FSL6.0.5.2module load FSL/6.0.5.2Analysis tools for FMRI, MRI and DTI brain imaging data
gcc13.0.1module load gcc/13.0.1Compiler - C / C++
glpk5.0module load glpk/5.0C library for solving large-scale linear programming (LP), mixed integer programming (MIP)
gurobi10.0.3module load gurobi/10/0/3Prescriptive analytics platform and a decision-making technology 
hdf51.10.1module load hdf5/1.10.1High performance data software library & file format
htslib

1.19

module load htslib/1.19

A C library for reading/writing high-throughput sequencing data
Intel oneAPI toolkit

various

module load intel-oneAPI-toolkit <library>

Core set of tools and libraries for developing high-performance, data-centric applications across diverse architectures. 
intel-parallel-studio

2020 Update 4

module load intel-parallel-studio/2020Intel Compiler 
julia1.5.3module load julia/1.5.3Programming Language
knitro13.2.0module load knitro/13.2.0Software package for solving large scale nonlinear mathematical optimization problems; short for "Nonlinear Interior point Trust Region Optimization"
lastz1.04.15module load lastz/1.04.15A program for aligning DNA sequences, a pairwise aligner.
LBPM22.08module load LBPM/22.08Model flow processes based on digital rock physics
leptonica

1.83.0

module load leptonica/1.83.0A ipedagogically-oriented open source library containing software that is broadly useful for image processing and image analysis 
libRadtran2.0.5modulel load libRadtran/2.0.5A library for radiative transfer
Mathematica13.2module load Mathematica/13.2Numerical Computing
Matlab2023amodule load matlab/2023aNumerical Computing
metis

5.1.0

module load metis/5.1.0

A set of serial programs for partitioning graphs, partitioning finite element meshes.
MUMPS

5.6.2

module load mumps/5.6.2

MUltifrontal Massively Parallel Sparse direct Solver
netcdf-fortran-intel4.5.3

module load netcdf-fortran-intel/4.5.3

Array Interface Library
netcdf/gcc4.7.4module load netcdf/gcc/64/gcc/64/4.7.4Array Interface Library
Nextflow

23.10.0

module load nextflow/23.10.0Enables scalable and reproducible scientific workflows using software containers.
occswV2022.3module load ocsswOcean Color Science Software, CLI version
octave5.2.0module load octaveInstalled on all compute nodes, start with 'octave'. Mathematics-oriented syntax with built-in 2D/3D plotting and visualization 
OpenFOAMv2206module load OpenFOAM/v2206Computational fluid dynamics.
openmpi4.1.5a1openmpi/gcc/64/4.1.5a1OpenMPI Compiler
Quantum Espresso7.2module load QE/7.2Quantum Espresso
R3.6.3module load R/3.6.3Programming Language
R4.3.1module load R/4.3.1Programming Language
samtools1.19module load samtools/1.19Suite of programs for interacting with high-throughput sequencing data

ScaLAPACK 

2.2.0module load scalapack/2.2.0Scalable Linear Algebra PACKage
Schrodinger2024-1module load schrodinger/2024-1Modeling, analysis and computational tasks

SeaDAS

9.0.1module load seadas/9.0.1

A comprehensive software package for the processing, display, analysis, and quality control of ocean color data. Requires XQuartz on a Mac or Mobaxterm on Windows.

Singularity3.7.1module load singularity/3.7.1

Run Docker-like containers

Stata18

module load stata/18

General-purpose statistical software
stopos0.93module load stoposCreate and manage computing tasks
tesseract5.3.1module load tesseractOCR
VCFTools0.1.17module load vcftools/0.1.17A set of tools written in Perl and C++ for working with VCF files
vim9.1module load vim/9.1vi improved test editor

Visual Studio Code Server


Not a moduleA server side Integrated Development Environment hosted on Ginsburg compute nodes
WIEN2kWIEN2k_21.1module load WIEN2k_21.1Perform electronic structure calculations of solids using density functional theory.
workbench1.5.0module load workbench/1.5.0Visualization and Discovery tool used to map neuro-imaging data 

Running brainiak in Anaconda Python

The Brain Imaging Analysis Kit is a package of Python modules useful for neuroscience, primarily focused on functional Magnetic Resonance Imaging (fMRI) analysis. We've managed to install it in a conda environment with anaconda/3-2022.05.  After loading the module run:

Code Block
conda activate /burg/opt/anaconda3-2022.05/envs/brainiak

You may have to initialize a shell first, e.g., conda init bash, note this will update your .bashrc file.

Installing Athena++

Athena++ radiation GRMHD code and adaptive mesh refinement (AMR) framework requires specific modules and can be tricky to compile. Load the following modules and respective versions and run the fllowinng commands :

Code Block
gcc/13.0.1 fftw3/openmpi/gcc/64/3.3.10 openmpi/gcc/64/4.1.5a1 hdf5p/hdf5p_1.14.2 anaconda
python configure.py --prob rt -b --flux hlld -mpi -hdf5 -fft --fftw_path /cm/shared/apps/fftw/openmpi/gcc/64/3.3.10 --hdf5_path /burg/opt/hdf5p-1.14.2

You can test an interactive session with salloc (not srun) with a sample input file:

mpirun -np 2 ../bin/athena -i athinput.rt3d
Setup complete, entering main loop...
cycle=0 time=0.0000000000000000e+00 dt=4.6376023638204525e-04
cycle=1 time=4.6376023638204525e-04 dt=4.6376640600387668e-04
cycle=2 time=9.2752664238592193e-04 dt=4.6377298288455065e-04
cycle=3 time=1.3912996252704725e-03 dt=4.6377996104450140e-04
cycle=4 time=1.8550795863149739e-03 dt=4.6378733390646439e-04

Running LBPM in a Singularity container from Nvidia's NGC Catalog

...

Some modifications are needed to get the water-flooding simulation to work correct. You can start with an interactive session via Slurm's srun salloc command and request a GPU as well as requesting at least 10 GB of memory.

srun --ptysalloc --mem=10gb -t 0-10:00 --gres=gpu:1  -A <your-account> /bin/bash

Update the wget commands as below:

export BENCHMARK_DIR=$PWD
wget https://gitlab.com/NVHPC/ngc-examples/-/raw/master/LBPM/2020.10/single-node/input.db
wget https://gitlab.com/NVHPC/ngc-examples/-/raw/master/LBPM/2020.10/single-node/run.sh
wget https://gitlab.com/NVHPC/ngc-examples/-/raw/master/LBPM/2020.10/single-node/mask_water_flooded_water_and_oil.raw.morphdrain.raw
chmod +x run.sh

...

The latest version of OpenMPI on Insomnia is  Ginsburg is 4.1.5a1, which is provided by Nvidia Mellanox and optimized for the MOFED stack, and is installed on each compute node to increase performance. You will receive the following warnings when using mpirun/mpiexec:

...

Code Block
srun -X -A <GROUP> --pty -t 0-0102:00 -X /bin/bash

OR, if you want a GPU node

srun -X -A <GROUP> --pty -t 0-0102:00 -X --gres=gpu:1 /bin/bash

Load the Singularity module:

Code Block
module load singularity

This next step will take a few minutes and will only need to be done the first time you launch this application. On subsequent sessions you skip this step as the container is downloaded:

Code Block
singularity pull --name rstudio.simg docker://rocker/rstudio:4.3.1

In order for RStudio to start in a browser via an interactive session you will need the IP address of the compute node. Note that the IP below will likely be different for you:

Code Block
IP_ADDR=`hostname -i`
echo $IP_ADDR
10.197.16.39

From RStudio 4.2 and later, some added security features require binding of a locally created database file to the database in the container. Don't forget to change the password.

Code Block
mkdir -p run var-lib-rstudio-server

printf 'provider=sqlite\ndirectory=/var/lib/rstudio-server\n' > database.conf

PASSWORD='CHANGEME' singularity exec \
   --bind run:/run,var-lib-rstudio-server:/var/lib/rstudio-server,database.conf:/etc/rstudio/database.conf \
   rstudio.simg \
   /usr/lib/rstudio-server/bin/rserver --auth-none=0 --auth-pam-helper-path=pam-helper --server-user=$USER

This will run the Rstudio server in a Singularity container.

PART 2:  Port forwarding and access from your browser

Now open another Terminal and start the RStudio rserver session using Port Forwarding to connect a local port on your computer to a remote one on Ginsburg.

Code Block
ssh -Nf -L 8787:10.197.16.39:8787 myUNI@burg.rcs.columbia.edu  
Note

NOTE FOR WINDOWS USERS:

If you are using Windows Subsystem for Linux (WSL/WSL2), then the above line work fine, BUT remember to forward the WSL session ports to the Windows browser. Set the System Browser as the WSL default and open 

The above Port Forwarding line works on Linux/Unix/MacOS, or any virtual machine running Linux. For Windows, we are assuming you are using PuTTY and the above line would be accomplished using the alternative steps below:

I.    Launch PuTTY

II.   In the "Session" category (the default screen when you open PuTTY), enter burg.rcs.columbia.edu in the "Host Name (or IP address)" field. 

III.   Ensure the "Port" field is set to 22, which is the default SSH port.       Optionally, you can enter a name in the "Saved Sessions" field and click "Save" to save these settings for future re-use

IV.   In the "Category" pane on the left, expand the "SSH" category by clicking the + sign next to it.

V.    Click on "Tunnels" under the "SSH" category.

VI.   In the "Source port" field, enter 8787.

VII.   In the "Destination" field, enter 10.197.16.39:8787.    (Remember,  10.197.16.39 is only an example IP, your actual one may be different)

VIII.   Click the "Add" button. You should see the forwarded port appear in the list box as L8787 10.197.16.39:8787.

IX.     Optional but recommended,   To save this configuration for future use, go back to the "Session" category, and enter a name for the session in the "Saved Sessions" field (e.g., <myuni>@burg.rcs.columbia.edu) and click the "Save" button.

X.      Click the "Open" button at the bottom of the PuTTY window and connect to burg.rcs.columbia.edu

Next, use a web browser to connect to to your locally forwarded port.

In any browser, e.g., Google Chrome, enter localhost:8787. The login screen for RStudio will appear and you enter your UNI (without the @columbia.edu) and whatever password you put from the above example (where it says CHANGME). If you need a session that last longer than an hour you can add --auth-timeout-minutes=0 --auth-stay-signed-in-days=30 to the end of the above singularity. command. A sample Slurm bash script is available at Rocker Project's Singularity page. If you run into an error such as [rserver] ERROR system error 98 (Address already in use), you can specify the port number as an additional option, e.g.,  --www-port 8788. The new command could look like this:

Code Block
mkdir -p run var-lib-rstudio-server

printf 'provider=sqlite\ndirectory=/var/lib/rstudio-server\n' > database.conf

PASSWORD='CHANGEME' singularity exec \
   --bind run:/run,var-lib-rstudio-server:/var/lib/rstudio-server,database.conf:/etc/rstudio/database.conf \
   rstudio.simg \
   /usr/lib/rstudio-server/bin/rserver --www-port 8788 --auth-none=0 --auth-pam-helper-path=pam-helper --server-user=$USER

***IMPORTANT NOTE***

When you are finished with RStudio remember to stop the Singularity session via ctl-c, and you can kill the SSH forwarding session by finding the process ID (PID) with the following command, noting that the PID is the second column:

Code Block
ps -ef | grep 8787
  503 82438     1   0 10:56AM ??         0:00.31 ssh -Nf -L 8787:10.197.16.39:8787 myUNI@burg.rcs.columbia.edu
kill -9 82438

Visual Studio Code Server

Note

A pre-existing Github account is now required to use the below instructions.

Visual Studio Code is an Integrated Development Environment that some like to use on their laptops. If you are familiar with that, the HPC has a server-side version hosted on the compute nodes (NOT the login nodes) for users to connect their local VS Code application on their laptop to Ginsburg and open files from their Ginsburg folder directly.  To use it, do the following:

After logging into Ginsburg, start an srun session to a CPU or GPU node of your choice:

Code Block
srun --pty -t 0-01:00  -A <ACCOUNT> /bin/bash

     OR, if you need a GPU

srun --pty -t 0-01:00 --gres=gpu:1 -A <ACCOUNT> /bin/bash

Then type

...

Code Block
* Visual Studio Code Server

*

* By using the software, you agree to

* the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and

* the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).

*

[2024-04-05 16:12:16] info Using Github for authentication, run `code tunnel user login --provider <provider>` option to change this.

To grant access to the server, please log into https://github.com/login/device and use code <###-###>

When you use the device login url, you will first get a page asking you to use your GitHub credentials to login.

After using your GitHub login, then you will get a page asking you to input the device code  given you above (represented as <###-###> here)

Next you will see a page requesting you to authorize VS Code studio's access permissions.

After that, when you use your local VS Code application on your computer, you will see a running ssh tunnel listed. Double click to connect to it. This can take a moment to finish.

...

gpu:1 /bin/bash


NEXT, a number of steps need to be done that have been collected into a simple script.

Type "RSserver.sh" (without the quotes) to execute the container setup script. For those interested, the script echos the commands it's running as it runs them.



Info
titleNOTE:

IF YOU GET AN ERROR  such as [rserver] ERROR system error 98 (Address already in use)

simply re-run the script and enter a different port number instead of the default of 8787 (the script will prompt you). Try 8788,  or 8789, etc



PART 2:  Port forwarding and access from your browser

The Rstudio server is now running in a container in one window.  Now you will open another window and create a Port Forwarding connection so your web browser will be able to connect to a port on your local machine, that forwards the connection to the remote container.


If you're on a Mac or Linux machine, open a second Terminal and start the RStudio rserver port forwarding with the below command:

Code Block
ssh -Nf -L 8787:10.197.16.39:8787 myUNI@burg.rcs.columbia.edu    <-- the "10.197.16.39" IP number is ONLY for demonstration. The above script will have given you the real IP you will be using.


Note

NOTE FOR WINDOWS USERS:

If you are using Windows Subsystem for Linux (WSL/WSL2), then the above line work fine, BUT remember to forward the WSL session ports to the Windows browser. Set the System Browser as the WSL default to open links.

Other Windows users instead need to refer to the port forwarding directions below.


The above Port Forwarding line works on Linux/Unix/MacOS, or any virtual machine running Linux. For Windows, we are assuming you are using PuTTY and the above line would be accomplished using the alternative steps below:

I.    Launch PuTTY

II.   In the "Session" category (the default screen when you open PuTTY), enter burg.rcs.columbia.edu in the "Host Name (or IP address)" field. 

III.   Ensure the "Port" field is set to 22, which is the default SSH port.       Optionally, you can enter a name in the "Saved Sessions" field and click "Save" to save these settings for future re-use

IV.   In the "Category" pane on the left, expand the "SSH" category by clicking the + sign next to it.

V.    Click on "Tunnels" under the "SSH" category.

VI.   In the "Source port" field, enter 8787.

VII.   In the "Destination" field, enter 10.197.16.39:8787.    (Remember,  10.197.16.39 is only an example IP, your actual one may be different)

VIII.   Click the "Add" button. You should see the forwarded port appear in the list box as L8787 10.197.16.39:8787.

IX.     Optional but recommended,   To save this configuration for future use, go back to the "Session" category, and enter a name for the session in the "Saved Sessions" field (e.g., <myuni>@burg.rcs.columbia.edu) and click the "Save" button.

X.      Click the "Open" button at the bottom of the PuTTY window and connect to burg.rcs.columbia.edu


Next, use a web browser to connect to to your locally forwarded port.

In any web browser, e.g., Google Chrome, Safari or Edge, enter localhost:8787 (or a different number after the colon, if you received an error above and had to choose another port).

The login screen for RStudio will appear and you enter your UNI (without the @columbia.edu) and whatever password you entered in the setup script.

A sample Slurm bash script is available at Rocker Project's Singularity page.


***IMPORTANT NOTE***

When you are finished with RStudio remember to stop the Singularity session via ctl-c, and you can kill the SSH forwarding session by finding the process ID (PID) with the following command, noting that the PID is the second column:

Code Block
ps -ef | grep 8787
  503 82438     1   0 10:56AM ??         0:00.31 ssh -Nf -L 8787:10.197.16.39:8787 myUNI@burg.rcs.columbia.edu
kill -9 82438



Visual Studio Code Server

Note

A pre-existing Github account is now required to use the below instructions. Create one first if you do not have one.


Visual Studio Code is an Integrated Development Environment that some like to use on their laptops. If you are familiar with that, the HPC has a server-side version hosted on the compute nodes (NOT the login nodes) for users to connect their local VS Code application on their laptop to Ginsburg and open files from their Ginsburg folder directly.  To use it, do the following:

After logging into Ginsburg, start an srun session to a CPU or GPU node of your choice:

Code Block
srun --pty -t 0-01:00  -A <ACCOUNT> /bin/bash

     OR, if you need a GPU

srun --pty -t 0-01:00 --gres=gpu:1 -A <ACCOUNT> /bin/bash


Then type

$ code tunnel

You will get a message like the one below:

Code Block
* Visual Studio Code Server

*

* By using the software, you agree to

* the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and

* the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).

*

[2024-04-05 16:12:16] info Using Github for authentication, run `code tunnel user login --provider <provider>` option to change this.

To grant access to the server, please log into https://github.com/login/device and use code <###-###>


When you use the device login url, you will first get a page asking you to use your GitHub credentials to login.

After using your GitHub login, then you will get a page asking you to input the device code  given you above (represented as <###-###> here)

Next you will see a page requesting you to authorize VS Code studio's access permissions.

After that, when you use your local VS Code application on your computer, you will see a running ssh tunnel listed. Double click to connect to it. This can take a moment to finish.

Once done, you will be able to open files in your Ginsburg folder the same as you do ones on your local computer.

Info

It's possible that the next time you try to use the tunnel you've just made, you may not be able to re-connect. The connection may keep trying endlessly, or finally fail with an error. This is because the srun session in the first step has placed you on another node different from the one where you created the tunnel. IF that happens, follow the steps below: 

1) In your local VSCode app (if you use a local copy of VS Code and not the browser-based window version), unregister/delete all of your existing tunnels. We’ll be making a new one.

2) Login to Ginsburg. Do not start an srun session yet.

3) Delete your invisible vscode settings folders in your home directory  (" rm -r .vscode* ")

4) Start an srun session as described above, type "code tunnel", and follow all of the directions as normal. Open VScode in the web browser first. It will take a moment as the new tunnel is created. If you only use VS Code in the browser, you are now done.

5) If you prefer to use a local copy of VS Code on your computer, you can now connect to this tunnel in it. WINDOWS USERS, additionally you may need to downgrade your local copy of Visual Studio Code to version 1.8.1 due to a bug existing as of this writing (8/28/2024).