{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"maxLevel":{"value":"2"}},"macroMetadata":{"macroId":{"value":"34f00d62-f138-4429-b8e4-6724321f50f3"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"52c25c3a-af4e-4104-b735-949e5a37a151"}},{"type":"paragraph","content":[{"text":"In order for the scripts in these examples to work, you will need to replace with your group's account name.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Hello World","type":"text"}]},{"type":"paragraph","content":[{"text":"This script will print \"Hello World\", sleep for 10 seconds, and then print the time and date. The output will be written to a file in your current directory.","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/sh\n# \n# Simple \"Hello World\" submit script for Slurm.\n#\n# Replace ACCOUNT with your account name before submitting.\n#\n#SBATCH --account=ACCOUNT # Replace ACCOUNT with your group account name\n#SBATCH --job-name=HelloWorld # The job name\n#SBATCH -c 1 # The number of cpu cores to use (up to 160 cores per server as hyperthreading is enabled)\n#SBATCH --time=0-0:30 # The time the job will take to run in D-HH:MM\n#SBATCH --mem-per-cpu=5G # The memory the job will use per cpu core\n\necho \"Hello World\"\nsleep 10\ndate\n\n# End of script\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Provided you saved this script in your home directory as \"helloworld.sh\", you would make it executable and submit it to the cluster for processing using the following commands:","type":"text"}]},{"type":"table","attrs":{"layout":"default","width":1800.0,"localId":"19f49051-58ce-43b3-a95b-d03ae4864dc5"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"chmod +x helloworld.sh","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"sbatch helloworld.sh","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"\"sbatch \" is the way you submit any script to the cluster for processing.","type":"text"}]},{"type":"paragraph","content":[{"text":"For more detailed information about SBATCH directives and submission scripts, see the section we have about ","type":"text"},{"text":"submitting jobs","type":"text","marks":[{"type":"strong"},{"type":"link","attrs":{"href":"https://columbiauniversity.atlassian.net/wiki/display/rcs/Insomnia+-+Submitting+Jobs"}}]},{"text":".","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Examples of some more advanced scripts follow below.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Running Precompiled Binaries","type":"text"}]},{"type":"paragraph","content":[{"text":"To submit a precompiled binary to run on Insomnia, the script will look just as it does in the Hello World example. The difference is that you will call your executable file instead of the shell commands \"echo\", \"sleep\", and \"date\".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"C, C++, Fortran MPI ","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"OneAPI","type":"text"}]},{"type":"paragraph","content":[{"text":"Insomnia supports OneAPI which is a highly optimized compiler that builds software with the highest performance. It also supports MPI for applications that require communication between multiple nodes. All the nodes on the cluster have Infiniband transport and that is the fabric that MPI jobs avail themselves of - which is another reason for a substantial boost of efficiency on the cluster.","type":"text"}]},{"type":"paragraph","content":[{"text":"To use OneAPI, you must load the module first:","type":"text"}]},{"type":"codeBlock","content":[{"text":"moddule load oneapi/hpctoolkit/hpctoolkit-2024.0.0\nmpiexec -bootstrap slurm ./myprogram\n","type":"text"}]},{"type":"paragraph","content":[{"text":"In order to take advantage of Insomnia architecture, your program should be (re)compiled on the cluster even if you used Intel for compiling it on another cluster. It is important to compile with the compiler provided by the module mentioned above. Note that you may have to set additional environment variables in order to successfully compile your program.","type":"text"}]},{"type":"paragraph","content":[{"text":"These are the locations of the C and Fortran compilers for OneAPI:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ module load oneapi/hpctoolkit/hpctoolkit-2024.0.0\n$ which mpicc /insomnia001/shared/apps/anaconda/2023.09/bin/mpicc\n\nFor Fortran:\n\n$ ml oneapi/hpctoolkit/ifort \n$ which ifort \n/insomnia001/shared/apps/oneapi/hpctoolkit/compiler/2024.0/bin/ifort ","type":"text"}]},{"type":"paragraph","content":[{"text":"For programs written in C, use mpicc in order to compile them:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ mpicc -o \n","type":"text"}]},{"type":"paragraph","content":[{"text":"The submit script below, named pi_mpi.sh, assumes that you have compiled a simple MPI program used to compute pi, (see ","type":"text"},{"text":"mpi_test.c","type":"text","marks":[{"type":"link","attrs":{"href":"https://columbiauniversity.atlassian.net/wiki/download/attachments/62135732/mpi_test.c?version=1&modificationDate=1351290549000&cacheVersion=1&api=v2"}}]},{"text":"), and created a binary called pi_mpi:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/sh\n\n#SBATCH -A ACCOUNT # Replace ACCOUNT with your group account name\n#SBATCH -N 2 # Number of nodes\n#SBATCH --mem-per-cpu=5800 # Default is 5800\n#SBATCH --time=0-0:30 # Runtime in D-HH:MM\n#SBATCH --ntasks-per-node=80 # Max 80 since Insomnia has 80 cores per node\n\nmodule load oneapi/hpctoolkit/hpctoolkit-2024.0.0\n\nmpiexec -bootstrap slurm ./pi_mpi\n\n# End of script\n","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":6},"content":[{"text":"Job Submission (i.e. how you actually send it to the cluster)","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ sbatch pi_mpi.sh\n","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"OpenMPI5","type":"text"}]},{"type":"paragraph","content":[{"text":"Insomnia also supports OpenMPI5 from the GNU family.","type":"text"}]},{"type":"paragraph","content":[{"text":"To use OpenMPI, you must load the openmpi module instead:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/sh\n\n#SBATCH -A ACCOUNT # Replace ACCOUNT with your account name\n#SBATCH -N 2\n#SBATCH --ntasks-per-node=80\n#SBATCH --time=0-0:30 # Runtime in D-HH:MM\n\nmodule load openmpi5\n\nmpiexec myprogram\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Your program must be compiled on the cluster. You can use the the module command as explained above to set your path so that the corresponding mpicc will be found. Note that you may have to set additional environment variables in order to successfully compile your program.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ module load openmpi5\n$ which mpicc \n/insomnia001/shared/apps/openmpi5/5.0.2/bin/mpicc ","type":"text"}]},{"type":"paragraph","content":[{"text":"Compile your program using mpicc. For programs written in C:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ mpicc -o \n","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Intel oneAPI with Hyperthreading options.","type":"text"}]},{"type":"paragraph","content":[{"text":"There are a few recommended ways to take advantage of hyper-threading while running an MPI application. Hyperthreading is enabled on all Insomnia compute/execute nodes.","type":"text"}]},{"type":"paragraph","content":[{"text":"1. To use all logical processors (including hyper-threaded cores) on a node, you can use the ","type":"text"},{"text":"`all`","type":"text","marks":[{"type":"code"}]},{"text":" option for ","type":"text"},{"text":"`I_MPI_PIN_PROCESSOR_LIST`","type":"text","marks":[{"type":"code"}]},{"text":". For example:","type":"text"}]},{"type":"paragraph","content":[{"text":"$ mpirun -genv I_MPI_PIN_PROCESSOR_LIST=all -n <# total processes> ./app","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"This will allow MPI processes to be placed on both physical cores and their hyper-threaded siblings.","type":"text"}]},{"type":"paragraph","content":[{"text":"2. If you want to use hyper-threading but avoid sharing resources between adjacent MPI processes, you can use the ","type":"text"},{"text":"`map=scatter`","type":"text","marks":[{"type":"code"}]},{"text":" option along with ","type":"text"},{"text":"`allcores`","type":"text","marks":[{"type":"code"}]},{"text":". This will scatter the processes across all physical cores first before utilizing the hyper-threaded siblings. For example:","type":"text"}]},{"type":"paragraph","content":[{"text":"$ mpirun -genv I_MPI_PIN_PROCESSOR_LIST=allcores:map=scatter -n <# total processes> ./app","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"This approach can potentially improve performance by reducing resource contention between processes that share the same physical core.","type":"text"}]},{"type":"paragraph","content":[{"text":"3. You can also specify a custom pinning granularity using the ","type":"text"},{"text":"`grain`","type":"text","marks":[{"type":"code"}]},{"text":" option. For example, if you want to pin two processes per physical core (one on the core and one on its hyper-threaded sibling), you can use:","type":"text"}]},{"type":"paragraph","content":[{"text":"$ mpirun -genv I_MPI_PIN_PROCESSOR_LIST=allcores:grain=2 -n <# total processes> ./app","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"This will create pinning cells with two logical processors each, allowing you to take advantage of hyper-threading while maintaining some level of resource isolation.","type":"text"}]},{"type":"paragraph","content":[{"text":"The optimal approach may depend on the specific characteristics of your MPI application, such as its resource usage patterns and sensitivity to resource contention. It's generally recommended to experiment with different pinning strategies and measure the performance to find the best configuration for your use case.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"GPU (CUDA C/C++)","type":"text"}]},{"type":"paragraph","content":[{"text":"As of March 1, 2024, the cluster currently includes ","type":"text"},{"text":"6 A40 GPUS ","type":"text","marks":[{"type":"strong"}]},{"text":"(2 GPU modules per server), ","type":"text"},{"text":"2402-gpu-node00[35-37]","type":"text","marks":[{"type":"strong"}]},{"text":". Three additional H100 GPU nodes are scheduled to arrive late in 2024.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"To use a GPU server you must specify the ","type":"text"},{"text":"--gres=gpu","type":"text","marks":[{"type":"strong"}]},{"text":" option in your submit request, followed by a colon and the number of GPU modules you require (with a maximum of 2 per server). ","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Request a gpu, specify this in your submit script. If the colon and number are omitted, as shown below, the scheduler will request 1 GPU module.","type":"text"}]},{"type":"codeBlock","content":[{"text":"#SBATCH --gres=gpu ","type":"text"}]},{"type":"paragraph","content":[{"text":"Not all applications have GPU support, but some, such as MATLAB, have built-in GPU support and can be configured to use GPUs.","type":"text"}]},{"type":"paragraph","content":[{"text":"To build your CUDA code and run it on the GPU modules you must first set your paths so that the Nvidia compiler can be found. Please note you must be logged into a GPU node to access these commands. To login interactively to a GPU node, run the following command, replacing with your account.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ srun --pty -t 0-01:00 --gres=gpu:1 -A /bin/bash\n","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"On Insomnia the CUDA module is automatically installed on the 3 GPU compute/execute nodes, ","type":"text"},{"text":"2402-gpu-node00[35-37]. ","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"HOWEVER, certain aspects, such as nvcc, are not loaded unless the explicit module load statement is used.","type":"text"}]},{"type":"paragraph","content":[{"text":"If you are compiling cuda code directly and need to compile your program using ","type":"text"},{"text":"nvcc","type":"text","marks":[{"type":"link","attrs":{"href":"http://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/"}}]},{"text":", below is an exmaple.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ module load cuda\n\n$ nvcc -o \n","type":"text"}]},{"type":"paragraph","content":[{"text":"You can compile ","type":"text"},{"text":"hello_world.cu","type":"text","marks":[{"type":"link","attrs":{"href":"https://columbiauniversity.atlassian.net/wiki/download/attachments/62138761/hello_world.cu?version=1&modificationDate=1479195540000&cacheVersion=1&api=v2"}}]},{"text":" sample code which can be built with the following command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ module load cuda\n\n$ nvcc -o hello_world hello_world.cu\n","type":"text"}]},{"type":"paragraph","content":[{"text":"For non-trivial code samples, refer to Nvidia's ","type":"text"},{"text":"CUDA Toolkit Documentation","type":"text","marks":[{"type":"link","attrs":{"href":"http://docs.nvidia.com/cuda/cuda-samples/#samples-reference"}}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"A Slurm script template, gpu.sh, that can be used to submit this job is shown below:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/sh\n#\n#SBATCH --account=ACCOUNT # The account name for the job.\n#SBATCH --job-name=HelloWorld # The job name.\n#SBATCH --gres=gpu:1 # Request 1 gpu (Up to 2 gpus per GPU node)\n#SBATCH -c 1 # The number of cpu cores to use.\n#SBATCH --time=0-01:00 # The time the job will take to run in D-HH:MM\n#SBATCH --mem-per-cpu=5gb # The memory the job will use per cpu core.\n\nmodule load cuda\n\n./hello_world\n\n# End of script \n","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Job submission (i.e. how you actually send the script off to the cluster)","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ sbatch gpu.sh\n","type":"text"}]},{"type":"paragraph","content":[{"text":"This program will print out \"Hello World!\" when run on a gpu server or print \"Hello Hello\" when no gpu module is found. ","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Apptainer (formerly called SINGULARITY) ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph","content":[{"text":"Apptainer","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}}]},{"text":" is a software tool that brings Docker-like containers and reproducibility to scientific computing and HPC. Apptainer has Docker container support and enables users to easily run different flavors of Linux with different software stacks. These containers provide a single universal on-ramp from the laptop, to HPC, to cloud.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph","content":[{"text":"Users can run Apptainer containers just as they run any other program on our HPC clusters. Example usage of Apptainer is listed below. For additional details on how to use Singularity, please contact us or refer to the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"Apptainer User Guide","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}},{"type":"link","attrs":{"href":"https://www.sylabs.io/guides/2.6/user-guide/index.html"}}]},{"text":".","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Downloading Pre-Built Containers","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}},{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Apptainer makes it easy to quickly deploy and use software stacks or new versions of software. Since Singularity has Docker support, users can simply pull existing Docker images from ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"Docker Hub","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}},{"type":"link","attrs":{"href":"https://hub.docker.com/"}}]},{"text":" or download docker images directly from software repositories that increasingly support the Docker format. ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"Apptainer Container Library","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}},{"type":"link","attrs":{"href":"https://cloud.sylabs.io/library"}}]},{"text":" also provides a number of additional containers.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"You can use the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"pull","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}},{"type":"link","attrs":{"href":"https://www.sylabs.io/guides/2.6/user-guide/appendix.html#pull-command"}}]},{"text":" command to download pre-built images from an external resource into your current working directory. The docker:// uri reference can be used to pull Docker images. Pulled Docker images will be automatically converted to the Apptainer container format. ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph","content":[{"text":"Apptainer is automatically loaded on any compute node, CPU or GPU, so first start an srun session to whichever type of node you want to work with:","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph"},{"type":"codeBlock","content":[{"text":"srun --pty -t 0-02:00 -A /bin/bash\n OR\nsrun --pty -t 0-02:00 --gres=gpu:1 -A /bin/bash","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"This example pulls the default Ubuntu docker image from docker hub.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"$ apptainer pull docker://ubuntu","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Running Apptainer Containers","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph","content":[{"text":"Here's an example of pulling the latest stable release of the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"Tensorflow Docker image","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}},{"type":"link","attrs":{"href":"https://www.tensorflow.org/install/docker"}}]},{"text":" and running it with Apptainer. (Note: these pre-built versions may not be optimized for use with our CPUs.)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"After starting an srun session, pull a docker image. This also converts the downloaded docker image to Apptainer format and saves it in your current working directory:","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"$ apptainer pull tensorflow.sif docker://tensorflow/tensorflow","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"The container tensorflow.sif in now your current directory.","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"Once you have download a container, you can run it interactively in a shell or in batch mode.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Apptainer - Interactive Shell ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph","content":[{"text":"The ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"shell","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}},{"type":"link","attrs":{"href":"https://www.sylabs.io/guides/2.6/user-guide/appendix.html#shell-command"}}]},{"text":" command allows you to spawn a new shell within your container and interact with it as though it were a small virtual machine:","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"$ apptainer shell tensorflow.sif","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Apptainer: Invoking an interactive shell within container...","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"From within the Apptainer shell, you will see the Apptainer prompt and can run the downloaded software. In this example, python is launched and tensorflow is loaded.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"codeBlock","content":[{"text":"Apptainer > python\n>>> import tensorflow as tf\n>>> print(tf.__version__)\n2.16.1\n>>> exit()","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"When done, you may exit the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"Apptainer","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":" interactive shell with the \"exit\" command.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"Apptainer> exit","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Apptainer: Executing Commands ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph","content":[{"text":"The ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"exec","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}},{"type":"link","attrs":{"href":"https://www.sylabs.io/guides/2.6/user-guide/appendix.html#exec-command"}}]},{"text":" command allows you to execute a custom command within a container by specifying the image file. This is the way to invoke commands in your job submission script.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"$ apptainer exec tensorflow.sif [command]","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"For example, to run python example above using the exec command:","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"$ apptainer exec tensorflow.sif python -c 'import tensorflow as tf; print(tf.__version__)'","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"2.16.1","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Apptainer: Running a Batch Job","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"paragraph","content":[{"text":"Below is an example of job submission script named ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"submit.sh","type":"text","marks":[{"type":"strong"}]},{"text":" that runs ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"Apptainer","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":". Note that you may need to specify the full path to the Apptainer image you wish to run.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph"},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n# Apptainer example submit script for Slurm.\n#\n# Replace with your account name before submitting.\n#\n#SBATCH -A # Set Account name\n#SBATCH --job-name=tensorflow # The job name\n#SBATCH -c 1 # Number of cores\n#SBATCH -t 0-0:30 # Runtime in D-HH:MM\n#SBATCH --mem-per-cpu=5gb # Memory per cpu core\n\napptainer exec tensorflow.sif python -c 'import tensorflow as tf; print(tf.__version__)'","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"Then submit the job to the scheduler. This example prints out the tensorflow version. After saving the above code into a file, make it executable and submit it to the cluster.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph","content":[{"text":"$ chmod +x submit.sh","type":"text"}]},{"type":"paragraph","content":[{"text":"$ sbatch submit.sh","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using MAKER in an Apptainer container","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph","content":[{"text":"MAKER is an easy-to-use genome annotation pipeline designed to be usable by small research groups with little bioinformatics experience. It has many dependencies, especially for Perl and using a container is a convenient way to have all the requirements in one place. The ","type":"text"},{"text":"BioContainers website maintains a apptainer","type":"text","marks":[{"type":"link","attrs":{"href":"https://biocontainers.pro/tools/maker"}}]},{"text":" container as of Dec. 2023, version 3.01.03. Here is a sample tutorial.","type":"text"}]},{"type":"paragraph","content":[{"text":"First start an srun session to a compute node.","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Note that the environment variable ","type":"text"},{"text":"LIBDIR","type":"text","marks":[{"type":"code"}]},{"text":" needs to be set and since ","type":"text"},{"text":"export","type":"text","marks":[{"type":"code"}]},{"text":" is a ","type":"text"},{"text":"Builtin command","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html"}}]},{"text":" that needs to be called via the ","type":"text"},{"text":"bash","type":"text","marks":[{"type":"code"}]},{"text":" command: ","type":"text"}]},{"type":"paragraph","content":[{"text":"apptainer run https://depot.galaxyproject.org/singularity/maker:3.01.03--pl5262h8f1cd36_2 bash -c 'export LIBDIR=/usr/local/share/RepeatMasker && maker --help'","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"You can ignore the warning:","type":"text"}]},{"type":"paragraph","content":[{"text":"Possible precedence issue with control flow operator at /usr/local/lib/site_perl/5.26.2/Bio/DB/IndexedBase.pm line 805.","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Using ","type":"text"},{"text":"example 03 legacy from the MAKER Tutorial for WGS Assembly and Annotation Winter School","type":"text","marks":[{"type":"link","attrs":{"href":"https://weatherby.genetics.utah.edu/MAKER/wiki/index.php/MAKER_Tutorial_for_WGS_Assembly_and_Annotation_Winter_School_2018#Merge.2FResolve_Legacy_Annotations"}}]},{"text":", you will be using the ","type":"text"},{"text":"model_gff","type":"text","marks":[{"type":"code"}]},{"text":" option to pass in legacy gene models. The full tutorial files can be ","type":"text"},{"text":"downloaded here","type":"text","marks":[{"type":"link","attrs":{"href":"http://weatherby.genetics.utah.edu/data/maker_tutorial.tgz"}}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"wget http://weatherby.genetics.utah.edu/data/maker_tutorial.tgz","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"tar -zxvf maker_tutorial.tgz","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"cd maker_tutorial/example_03_legacy","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"apptainer run https://depot.galaxyproject.org/singularity/maker:3.01.03--pl5262h8f1cd36_2 bash -c 'export LIBDIR=/usr/local/share/RepeatMasker && maker -CTL'","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"cp opts.txt maker_opts.ctl (Type Y to overwrite the existing file)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"nano maker_opts.ctl (This is using the nano text editor to add the text in the next step).","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Add a label:","type":"text"}]},{"type":"paragraph","content":[{"text":"est=pyu_est.fasta:hypoxia","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"Write the file back to disk and exit nano","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"ctl-o ","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"ctl-x","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"apptainer run https://depot.galaxyproject.org/singularity/maker:3.01.03--pl5262h8f1cd36_2 bash -c \"export LIBDIR=/usr/local/share/RepeatMasker && maker\"","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Possible precedence issue with control flow operator at /usr/local/lib/site_perl/5.26.2/Bio/DB/IndexedBase.pm line 805.","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"STATUS: Parsing control files...","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"STATUS: Processing and indexing input FASTA files...","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"STATUS: Setting up database for any GFF3 input...","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"A data structure will be created for you at:","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"/burg/home/you/maker_tutorial/example_03_legacy/legacy_contig.maker.output/legacy_contig_datastore","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"To access files for individual sequences use the datastore index:","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"/burg/home/you/maker_tutorial/example_03_legacy/legacy_contig.maker.output/legacy_contig_master_datastore_index.log","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"STATUS: Now running MAKER...","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"examining contents of the fasta file and run log","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"[...]","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Maker is now finished!!!","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"Start_time: 1701293886","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"End_time: 1701293922","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Elapsed: 36","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"To use Maker with OpenMPI, e.g., requesting 8 CPU ","type":"text"},{"text":"ntasks","type":"text","marks":[{"type":"link","attrs":{"href":"https://slurm.schedmd.com/sbatch.html#OPT_ntasks"}}]},{"text":" (which are ","type":"text"},{"text":"processes that a job executes in parallel in one or more nodes","type":"text","marks":[{"type":"textColor","attrs":{"color":"#0c0d0e"}},{"type":"link","attrs":{"href":"https://stackoverflow.com/a/73828155/7606730"}}]},{"text":")","type":"text","marks":[{"type":"textColor","attrs":{"color":"#0c0d0e"}}]},{"text":", you can use the following suggested options, which will help reduce warnings. Start with an interactive session using the ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":" command and increase the requested memory as needed:","type":"text"}]},{"type":"paragraph","content":[{"text":"salloc --ntasks=8 --account=test --mem=50GB srun -n1 -N1 --mem-per-cpu=0 --gres=NONE --pty --preserve-env --mpi=none $SHELL","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"module load openmpi/gcc/64/4.1.5a1 apptainer","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"mpirun -np 2 --mca btl '^openib' --mca orte_base_help_aggregate 0 apptainer run https://depot.galaxyproject.org/singularity/maker:3.01.03--pl5262h8f1cd36_2 bash -c \"export LIBDIR=/usr/local/share/RepeatMasker && maker\"","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Additionally ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"samtools","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}},{"type":"link","attrs":{"href":"https://www.htslib.org/"}}]},{"text":" (used for reading/writing/editing/indexing/viewing SAM/BAM/CRAM format) is available in the container:","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph","content":[{"text":"apptainer> samtools --version","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"samtools 1.7","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Using htslib 1.7-2","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Copyright (C) 2018 Genome Research Ltd.","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Note, if you are testing ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"maker","type":"text","marks":[{"type":"code"}]},{"text":" and kill jobs/processes look out for ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":".NFSLock ","type":"text","marks":[{"type":"link","attrs":{"href":"https://groups.google.com/g/maker-devel/c/9Aw7CS5oep0"}}]},{"text":"files","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}},{"type":"link","attrs":{"href":"https://groups.google.com/g/maker-devel/c/9Aw7CS5oep0"}}]},{"text":", which you will likely need to delete for subsequent runs of ","type":"text"},{"text":"maker","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"},{"text":" You will need to use the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"-a","type":"text","marks":[{"type":"code"}]},{"text":" option with ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"ls","type":"text","marks":[{"type":"code"}]},{"text":" as files that start with a dot/period are hidden from the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"ls","type":"text","marks":[{"type":"code"}]},{"text":" command by default.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using GATK in an Apptainer container","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph","content":[{"text":"GATK, the Genome Analysis Toolkit has several dependencies and can run inside a container. Here is a sample tutorial: ","type":"text"}]},{"type":"paragraph","content":[{"text":"module load apptainer","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"apptainer pull docker://broadinstitute/gatk","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Using ","type":"text"},{"text":"Step 4","type":"text","marks":[{"type":"link","attrs":{"href":"https://gatk.broadinstitute.org/hc/en-us/articles/360041320571--How-to-Install-all-software-packages-required-to-follow-the-GATK-Best-Practices"}}]},{"text":" at the GATK Best Practices page, download the suggested .bam file:","type":"text"}]},{"type":"paragraph","content":[{"text":"wget https://raw.githubusercontent.com/broadinstitute/gatk/master/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.bam","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Then run:","type":"text"}]},{"type":"paragraph","content":[{"text":"apptainer shell gatk_latest.sif","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"In the container run:","type":"text"}]},{"type":"paragraph","content":[{"text":"gatk PrintReads -I NA12878.chr17_69k_70k.dictFix.bam -O output.bam","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"The output should end with (timestamp removed for clarity):","type":"text"}]},{"type":"paragraph","content":[{"text":"INFO PrintReads - Done initializing engine","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"INFO ProgressMeter - Starting traversal","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"NFO PrintReads - 0 read(s) filtered by: WellformedReadFilter","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"NFO ProgressMeter - unmapped 0.0 493 896363.6","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"NFO ProgressMeter - Traversal complete. Processed 493 total reads in 0.0 minutes.","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"INFO PrintReads - Shutting down engine","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"org.broadinstitute.hellbender.tools.PrintReads done. Elapsed time: 0.00 minutes.","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Runtime.totalMemory()=285212672","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"To quit the container press ","type":"text"},{"text":"ctl-D","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"These 2 files were created:","type":"text"}]},{"type":"paragraph","content":[{"text":"-rw-r--r-- 1 rk3199 user 128 Nov 28 16:48 output.bai","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"-rw-r--r-- 1 rk3199 user 62571 Nov 28 16:48 output.bam","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using GeoChemFoam in an Apptainer container","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph","content":[{"text":"GeoChemFoam","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/GeoChemFoam"}}]},{"text":" is open source code, based on the OpenFoam CFD toolbox ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"developed at the Institute of GeoEnergy Engineering, Heriot-Watt University","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}},{"type":"link","attrs":{"href":"https://www.julienmaes.com/geochemfoam"}}]},{"text":". ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph","content":[{"text":"Some additional steps/tweaks are needed to get all of the features working. For this tutorial we assume ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"GeoChemFoam","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/GeoChemFoam"}}]},{"text":" version 5.1 and use the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"Test Case 01 Species transport","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}},{"type":"link","attrs":{"href":"https://github.com/GeoChemFoam/GeoChemFoam/wiki/Test-Case-01----Species-transport"}}]},{"text":" in a Ketton Micro-CT image tutorial","type":"text","marks":[{"type":"textColor","attrs":{"color":"#24292f"}},{"type":"link","attrs":{"href":"https://github.com/GeoChemFoam/GeoChemFoam/wiki/Test-Case-01----Species-transport"}}]},{"text":". You can choose your version of Anaconda Python. Note there are some changes in the tutorial from earlier versions of GeoChemFoam, e.g., 4.8. For this tutorial we assume the name of the Apptainer container is ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"geochemfoam-5.1.sif","type":"text","marks":[{"type":"code"}]},{"text":". and we'll use an interactive job with ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":", to request 8 ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"--ntasks","type":"text","marks":[{"type":"code"}]},{"text":" for use with OpenMPI in the Ketton tutorial. Note that Apptainer is installed on all compute nodes so there is no need to load a module for it. Also note that copy/pasting these commands may not work, so it's best to type these commands out. Note that using multiple nodes with either with ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"-N","type":"text","marks":[{"type":"code"}]},{"text":"/","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"--nodes","type":"text","marks":[{"type":"code"}]},{"text":"= or adding ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"-c","type":"text","marks":[{"type":"code"}]},{"text":"/","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"--cpus-per-task ","type":"text","marks":[{"type":"code"}]},{"text":"in your ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"SBATCH","type":"text","marks":[{"type":"code"}]},{"text":" script will not work and result in an error: \"A","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"n ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun.","type":"text","marks":[{"type":"code"}]},{"text":"\" ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph","content":[{"text":"salloc -t 0-01:0:00 --ntasks=8 --mem=12gb ","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"module load mpi/openmpi-x86_64","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"apptainer ","type":"text"},{"text":"offers a few ways","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.sylabs.io/guides/3.7/user-guide/environment_and_metadata.html"}}]},{"text":" to set/pass environment variables, we'll use APPTAINER","type":"text"},{"text":"ENV_PREPEND_PATH","type":"text","marks":[{"type":"code"}]},{"text":" as the container needs to know where the ","type":"text"},{"text":"srun","type":"text","marks":[{"type":"code"}]},{"text":" command is.","type":"text"}]},{"type":"paragraph","content":[{"text":"export APPTAINERENV_PREPEND_PATH=$PATH","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"apptainer shell --bind /path/to/your-directory/runs:/home/gcfoam/works/GeoChemFoam-5.1/runs:rw geochemfoam-5.1_latest.sif","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"--bind","type":"text","marks":[{"type":"code"}]},{"text":" connects whatever is to the left of the colon, in this case, a new directory called '","type":"text"},{"text":"runs","type":"text","marks":[{"type":"code"}]},{"text":"'. The right side of the colon is a path that exists in the container. ","type":"text"},{"text":"rw","type":"text","marks":[{"type":"code"}]},{"text":" is read/write","type":"text"}]},{"type":"paragraph","content":[{"text":"All files written to the ","type":"text"},{"text":"/runs","type":"text","marks":[{"type":"code"}]},{"text":" directory will remain and that's the only directory that files can be written to when the container is bound. ","type":"text"}]},{"type":"paragraph","content":[{"text":"Change the value of","type":"text"},{"text":" $HOME","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"paragraph","content":[{"text":"export HOME=/home/gcfoam","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Copy the ","type":"text"},{"text":"multiSpeciesTransportFoam","type":"text","marks":[{"type":"code"}]},{"text":" directory from","type":"text"},{"text":" /home/gcfoam/works/GeoChemFoam-5.1/tutorials/transport","type":"text","marks":[{"type":"code"}]},{"text":" to ","type":"text"},{"text":"/path/to/your-directory/runs/","type":"text","marks":[{"type":"code"}]},{"text":", e.g., ","type":"text"}]},{"type":"paragraph","content":[{"text":"cp -a /home/gcfoam/works/GeoChemFoam-5.1/tutorials/transport /path/to/your-directory/runs/","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Install 5 Python libraries using Python from within the container. Specify where to install these (","type":"text"},{"text":"-t","type":"text","marks":[{"type":"code"}]},{"text":" option as follows is one way).","type":"text"}]},{"type":"paragraph","content":[{"text":"/usr/bin/pip3 install -t /home/gcfoam/works/GeoChemFoam-5.1/runs matplotlib numpy scikit-image numpy-stl h5py","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Note this will populate the ","type":"text"},{"text":"~/runs ","type":"text","marks":[{"type":"code"}]},{"text":"directory where the results from the tutorial are so you may want to put these in a different directory.","type":"text"}]},{"type":"paragraph","content":[{"text":"Set 2 additional environment variables:","type":"text"}]},{"type":"paragraph","content":[{"text":"export PYTHONPATH=/home/gcfoam/works/GeoChemFoam-5.1/runs","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"export MPLCONFIGDIR=/home/gcfoam/works/GeoChemFoam-5.1/runs","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Source the ","type":"text"},{"text":".bashrc","type":"text","marks":[{"type":"code"}]},{"text":" file in the container:","type":"text"}]},{"type":"paragraph","content":[{"text":"source $HOME/works/GeoChemFoam-5.1/etc/bashrc","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Run the scripts in the tutorial in your","type":"text"},{"text":" ~/runs","type":"text","marks":[{"type":"code"}]},{"text":" directory.","type":"text"}]},{"type":"paragraph","content":[{"text":"cd transport/multiSpeciesTransportFoam/Ketton/","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"./createMesh.sh","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"To avoid seeing a couple of warnings from OpenMPI, you can add the following options to the ","type":"text"},{"text":"mpirun","type":"text","marks":[{"type":"code"}]},{"text":" command in the scripts (using ","type":"text"},{"text":"vi","type":"text","marks":[{"type":"code"}]},{"text":") that use them: ","type":"text"},{"text":"--mca btl '^openib' --mca orte_base_help_aggregate 0","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"For the next script, ","type":"text"},{"text":"runSnappyHexMesh.sh,","type":"text","marks":[{"type":"code"}]},{"text":" that would change as follows:","type":"text"}]},{"type":"paragraph","content":[{"text":"mpirun -np $NP --mca btl '^openib' --mca orte_base_help_aggregate 0 snappyHexMesh -overwrite -parallel > snappyHexMesh.out","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"./runSnappyHexMesh.sh","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"./initCaseFlow.sh","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"The next two scripts have ","type":"text"},{"text":"mpirun","type":"text","marks":[{"type":"code"}]},{"text":", which you can add","type":"text"},{"text":" --mca btl '^openib' --mca orte_base_help_aggregate 0","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"./runCaseFlow.sh","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"Check the ","type":"text"},{"text":"simpleFoamFlow.out","type":"text","marks":[{"type":"code"}]},{"text":" file for '","type":"text"},{"text":"SIMPLE solution converged in iterations","type":"text","marks":[{"type":"code"}]},{"text":"'","type":"text"},{"type":"hardBreak"},{"text":"./processFlow.sh","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"cat poroPerm.csv","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"time poro perm Lpore Re UD","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0 0.133986 5.71129e-12 1.84664e-05 0.0220027 0.00239467","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"./initCaseTransport.sh","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"The next two scripts contain ","type":"text"},{"text":"mpirun","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"paragraph","content":[{"text":"./runCaseTransport.sh","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"./processTransport.sh ","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"cat A_Conc.csv B_Conc.csv","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"time C","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0 0","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.01 0.274137","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.02 0.432351","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.03 0.554637","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.04 0.652788","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.05 0.730438","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.06 0.791012","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.07 0.838068","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.08 0.87449","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.09 0.90264","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.1 0.924441","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"time C","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0 0","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.01 0.126229","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.02 0.239035","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.03 0.345767","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.04 0.443858","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.05 0.533867","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.06 0.613993","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.07 0.681486","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.08 0.737184","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.09 0.782178","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"0.1 0.818111","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"To run the tutorial again make sure to run one of the delete scripts, e.g., ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"deleteAll.sh","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"For additional details on how to use apptainer, please contact us or refer to the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"apptainer User Guide","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1155cc"}},{"type":"link","attrs":{"href":"https://sylabs.io/docs/#doc-singularity-3.7"}}]},{"text":".","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using Couenne in an Apptainer container","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]}]},{"type":"paragraph","content":[{"text":"Couenne","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}},{"type":"link","attrs":{"href":"https://github.com/coin-or/Couenne"}}]},{"text":" (Convex Over and Under ENvelopes for Nonlinear Estimation) is a branch&bound algorithm to solve Mixed-Integer Nonlinear Programming (MINLP) problems. It includes a suite of programs with several dependencies. Fortunately there is a ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"Docker container","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}},{"type":"link","attrs":{"href":"https://hub.docker.com/r/coinor/coin-or-optimization-suite"}}]},{"text":" which can be used to access these programs, e.g., bonmin, couenne, Ipopt, Cgl, and Cbc, via apptainer. You can use ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":"these sample","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1f2328"}},{"type":"link","attrs":{"href":"https://github.com/cvanaret/Uno/tree/main/examples"}}]},{"text":" .nl files","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}},{"type":"link","attrs":{"href":"https://github.com/cvanaret/Uno/tree/main/examples"}}]},{"text":" to test","type":"text","marks":[{"type":"textColor","attrs":{"color":"#222222"}}]},{"text":" with Couenne.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#1f2328"}}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"apptainer pull docker://coinor/coin-or-optimization-suite ","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"apptainer shell coin-or-optimization-suite_latest.sif ","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"apptainer> couenne hs015.nl ","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Couenne 0.5 -- an Open-Source solver for Mixed Integer Nonlinear Optimization","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Mailing list: couenne@list.coin-or.org","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Instructions: http://www.coin-or.org/Couenne","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"NLP0012I ","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":" Num Status Obj It time Location","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"NLP0014I 1 OPT 306.49998 22 0.004007","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Couenne: new cutoff value 3.0649997900e+02 (0.009883 seconds)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Loaded instance \"hs015.nl\"","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Constraints: 2","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Variables: 2 (0 integer)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Auxiliaries: 8 (0 integer)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"Coin0506I Presolve 29 (-1) rows, 9 (-1) columns and 64 (-2) elements","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Clp0006I 0 Obj 0.25 Primal inf 473.75936 (14)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Clp0006I 13 Obj 0.31728151","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Clp0000I Optimal - objective value 0.31728151","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Clp0032I Optimal objective 0.3172815065 - 13 iterations time 0.002, Presolve 0.00","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Clp0000I Optimal - objective value 0.31728151","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Cbc0012I Integer solution of 306.49998 found by Couenne Rounding NLP after 0 iterations and 0 nodes (0.00 seconds)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"NLP Heuristic: NLP0014I 2 OPT 306.49998 5 0.001228","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"solution found, obj. 306.5","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Clp0000I Optimal - objective value 0.31728151","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Optimality Based BT: 3 improved bounds","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Probing: 2 improved bounds","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Cbc0031I 1 added rows had average density of 2","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Cbc0013I At root node, 4 cuts changed objective from 0.31728151 to 306.49998 in 1 passes","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Cbc0014I Cut generator 0 (Couenne convexifier cuts) - 4 row cuts average 2.0 elements, 3 column cuts (3 active)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Cbc0001I Search completed - best objective 306.4999790004336, took 0 iterations and 0 nodes (0.00 seconds)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"couenne: Optimal","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":" \"Finished\"","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"Linearization cuts added at root node: 30","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Linearization cuts added in total: 30 (separation time: 2.4e-05s)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Total solve time: 0.003242s (0.003242s in branch-and-bound)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Lower bound: 306.5","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Upper bound: 306.5 (gap: 0.00%)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Branch-and-bound nodes: 0","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Performance of FBBT: 2.8e-05s, 4 runs. fix: 0 shrnk: 0.000103838 ubd: 2.75 2ubd: 0.5 infeas: 0","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Performance of OBBT: 0.000742s, 1 runs. fix: 0 shrnk: 6.70203 ubd: 0 2ubd: 0 infeas: 0","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Example of R run","type":"text"}]},{"type":"paragraph","content":[{"text":"For this example, the R code below is used to generate a graph ''Rplot.pdf'' of a discrete Delta-hedging of a call. It hedges along a path and repeats over many paths. There are two R files required:","type":"text"}]},{"type":"paragraph","content":[{"text":"hedge.R","type":"text","marks":[{"type":"link","attrs":{"href":"https://columbiauniversity.atlassian.net/wiki/download/attachments/62135816/hedge.R.txt?version=1&modificationDate=1351284524000&cacheVersion=1&api=v2"}}]}]},{"type":"paragraph","content":[{"text":"BlackScholesFormula.R","type":"text","marks":[{"type":"link","attrs":{"href":"https://columbiauniversity.atlassian.net/wiki/download/attachments/62135816/blackscholesformula.R.txt?version=1&modificationDate=1351284673000&cacheVersion=1&api=v2"}}]}]},{"type":"paragraph","content":[{"text":"A Slurm script, hedge.sh, that can be used to submit this job is presented below:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/sh\n#hedge.sh\n#Slurm script to run R program that generates graph of discrete Delta-hedging call\n#\n#SBATCH -A ACCOUNT # Replace ACCOUNT with your group account name \n#SBATCH -J DeltaHedge # The job name\n#SBATCH -c 4 # The number of cpu cores to use. Max 32.\n#SBATCH -t 0-0:30 # Runtime in D-HH:MM\n#SBATCH --mem-per-cpu 5gb # The memory the job will use per cpu core\n\nmodule load R\n\n#Command to execute R code\nR CMD BATCH --no-save --vanilla hedge.R routput\n\n# End of script\n","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Batch queue submission","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ sbatch hedge.sh\n","type":"text"}]},{"type":"paragraph","content":[{"text":"This program will leave several files in the output directory: slurm-.out, Rplots.pdf, and routput (the first one will be empty).","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Running LAMMPS in an Apptainer Container with a GPU","type":"text"}]},{"type":"paragraph","content":[{"text":"Nvidia provides prebuilt apptainer LAMMPS (Large-scale Atomic/Molecular Massively Parallel Simulator) GPU-enabled containers in its ","type":"text"},{"text":"New General Catalogue","type":"text","marks":[{"type":"link","attrs":{"href":"https://catalog.ngc.nvidia.com/orgs/hpc/containers/lammps"}}]},{"text":" (NGC). Using the tutorial provided you can use:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ docker://nvcr.io/hpc/lammps:29Sep2021","type":"text"}]},{"type":"paragraph","content":[{"text":"Here is a modified tutorial for Insomnia, starting with an interactive session requesting one GPU, 10 GB of memory and note the extra option for the wget commands:","type":"text"}]},{"type":"codeBlock","content":[{"text":"srun --pty --mem=10gb -t 0-2:00 --gres=gpu:1 -A /bin/bash\nmodule load apptainer/3.7.1\nwget https://lammps.sandia.gov/inputs/in.lj.txt --no-check-certificate\nexport BENCHMARK_DIR=$PWD\ncd $BENCHMARK_DIR\nwget https://gitlab.com/NVHPC/ngc-examples/-/raw/master/lammps/single-node/run_lammps.sh --no-check-certificate \nchmod +x run_lammps.sh\ncd $BENCHMARK_DIR\napptainer run --nv -B $PWD:/host_pwd --pwd /host_pwd docker://nvcr.io/hpc/lammps:29Sep2021 ./run_lammps.sh","type":"text"}]},{"type":"paragraph","content":[{"text":"You should start seeing this response:","type":"text"}]},{"type":"codeBlock","content":[{"text":"Running Lennard Jones 8x4x8 example on 1 GPUS...\nLAMMPS (29 Sep 2021)\nKOKKOS mode is enabled (src/KOKKOS/kokkos.cpp:97)\n will use up to 1 GPU(s) per node\n using 1 OpenMP thread(s) per MPI task\nLattice spacing in x,y,z = 1.6795962 1.6795962 1.6795962","type":"text"}]},{"type":"paragraph","content":[{"text":"You can combine the commands for use with directly with a Slurm interactive session as suggested in the original tutorial","type":"text"}]},{"type":"codeBlock","content":[{"text":"srun --pty --mem=10gb -t 0-4:00 --gres=gpu:1 -A -v --mpi=pmi2 -c 4 apptainer run --nv -B $PWD:/host_pwd --pwd /host_pwd docker://nvcr.io/hpc/lammps:29Sep2021 ./run_lammps.sh","type":"text"}]},{"type":"paragraph","content":[{"text":"The same container can be used for non-GPU jobs but still requires requesting a GPU node. You can test with a tutorial that provides ","type":"text"},{"text":"polymer_plus_bridges.lam","type":"text","marks":[{"type":"link","attrs":{"href":"https://cbrackley.github.io/simple_lammps_tutorial/"}}]},{"text":". Download the .lam and initial configuration files. With srun, -v will show more verbose output.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"wget https://cbrackley.github.io/simple_lammps_tutorial/lammps_tutorial/tutorial5/polymer_plus_bridges.lam\nwget https://cbrackley.github.io/simple_lammps_tutorial/lammps_tutorial/tutorial5/initial_configuration.txt\nsrun --pty --mem=10gb -t 0-4:00 --gres=gpu:1 -A -v --mpi=pmi2 apptainer exec --nv -B $PWD:/host_pwd --pwd /host_pwd docker://nvcr.io/hpc/lammps:29Sep2021 mpirun lmp -in polymer_plus_bridges.lam\n[...]\nReading data file ...\n orthogonal box = (-25.000000 -25.000000 -25.000000) to (25.000000 25.000000 25.000000)\n 1 by 1 by 1 MPI processor grid\n reading atoms ...\n 220 atoms\n reading velocities ...\n 220 velocities\n scanning bonds ...\n 1 = max bonds/atom\n scanning angles ...\n 1 = max angles/atom\n reading bonds ...\n 199 bonds\n reading angles ...\n 198 angles\nFinding 1-2 1-3 1-4 neighbors ...","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Installing R Packages locally on Insomnia","type":"text"}]},{"type":"paragraph","content":[{"text":"HPC users can Install R packages locally in their home directory or group's scratch space. The below guide will use a User's group scratch space.","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"LOCAL INSTALLATION","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"After logging into Insomnia and using srun to get a ","type":"text"},{"text":"GPU","type":"text","marks":[{"type":"strong"}]},{"text":" node, start R. It is automatically there on ","type":"text"},{"text":"GPU node","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ srun --pty -t 0-01:00 --gres=gpu:1 -A /bin/bash\n\n$ R","type":"text"}]},{"type":"paragraph","content":[{"text":"You can see the default library paths (where R looks for packages) by calling .libPaths():","type":"text"}]},{"type":"codeBlock","content":[{"text":"> .libPaths() \n[1] \"/usr/lib64/R/library\" \"/usr/share/R/library\" ","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"These paths are all read-only, and so you cannot install packages to them. To fix this, we will tell R to look in additional places for packages.","type":"text"}]},{"type":"paragraph","content":[{"text":"Exit R and create a directory rpackages in /insomnia001/depts//users//","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ mkdir -p /insomnia001/depts//users//rpackages\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Go back into R and add this path to .libPaths()","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ R\n> .libPaths(\"/insomnia001/depts//users//rpackages\")\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Call .libPaths() to make sure the path has been added","type":"text"}]},{"type":"codeBlock","content":[{"text":"> .libPaths()\n[1] \"/insomnia001/depts//users//rpackages\"\n[2] \"/usr/lib64/R/library\" \n[3] \"/usr/share/R/library\" ","type":"text"}]},{"type":"paragraph","content":[{"text":"To install a package, such as the \"sm\" package, tell R to put the package in your newly created local library:","type":"text"}]},{"type":"codeBlock","content":[{"text":"> install.packages(\"sm\", lib=\"/insomnia001/depts//users//rpackages\")\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Select appropriate mirror and follow install instructions.","type":"text"}]},{"type":"paragraph","content":[{"text":"Test to see if package can be called:","type":"text"}]},{"type":"codeBlock","content":[{"text":"> library(sm) \nPackage 'sm', version 2.2-6.0: type help(sm) for summary information ","type":"text"}]},{"type":"paragraph","content":[{"text":"In order to access this library from your programs, ","type":"text"},{"text":"make sure","type":"text","marks":[{"type":"strong"}]},{"text":" you add the following line to the top of every program:","type":"text"}]},{"type":"codeBlock","content":[{"text":".libPaths(\"/insomnia001/depts//users//rpackages/\")\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Since R will know where to look for libraries, a call to library(sm) will be successful (however, this line is not necessary per se for the install.packages(...) call, as the directory is already specified in it).","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"MATLAB","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"MATLAB atlab (single thread)","type":"text"}]},{"type":"paragraph","content":[{"text":"The file linked below is a MATLAB M-file containing a single function, simPoissGLM, that takes one argument (lambda).","type":"text"}]},{"type":"paragraph","content":[{"text":"simPoissGLM.m","type":"text","marks":[{"type":"link","attrs":{"href":"https://columbiauniversity.atlassian.net/wiki/download/attachments/62135832/simpoissglm.m.txt?version=1&modificationDate=1351290771000&cacheVersion=1&api=v2"}}]}]},{"type":"paragraph","content":[{"text":"A Slurm script, simpoiss.sh, that can be used to submit this job is presented below.","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/sh\n#\n# Simple MATLAB submit script for Slurm.\n#\n#\n#SBATCH -A ACCOUNT # Replace ACCOUNT with your group account name \n#SBATCH -J SimpleMLJob # The job name\n#SBATCH -c 1 # Number of cores to use (max 32)\n#SBATCH -t 0-0:30 # Runtime in D-HH:MM\n#SBATCH --mem-per-cpu=5G # The memory the job will use per cpu core\n\nmodule load MATLAB echo \"Launching a MATLAB run\"\ndate\n\n#define parameter lambda\nLAMBDA=10\n\n#Command to execute MATLAB code\nMATLAB -nosplash -nodisplay -nodesktop -r \"simPoissGLM($LAMBDA)\" # > matoutfile\n\n# End of script\n\n","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Batch queue submission","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ sbatch simpoiss.sh\n","type":"text"}]},{"type":"paragraph","content":[{"text":"This program will leave several files in the output directory: slurm-.out, out.mat, and matoutfile.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"MATLAB with Parallel Computing Toolbox","type":"text"}]},{"type":"paragraph","content":[{"text":"Parallel Computing Toolbox (PCT)™, ","type":"text"},{"text":"formerly known as Distributed Computing Engine","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.mathworks.com/matlabcentral/answers/217639-what-is-the-difference-between-the-distrib_comp_engine-and-distrib_computing_toolbox-licenses"}}]},{"text":", is installed within MATLAB. It lets you solve computationally and data-intensive problems using multicore processors, but limited to one compute/execute node. The compute nodes on Insomnia have 32 CPUs, over 2 sockets, with 16 cores per socket. In order to use PCT you will have to incorporate MATLAB functions such as ","type":"text"},{"text":"distributed","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.mathworks.com/help/releases/R2020b/parallel-computing/distributed.distributed.html"}}]},{"text":" or ","type":"text"},{"text":"parfor","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.mathworks.com/help/releases/R2020b/parallel-computing/parallel-for-loops-parfor.html?s_tid=CRUX_lftnav"}}]},{"text":". Note in order to use all 32 CPUs the best practice is to use the ","type":"text"},{"text":"--exclusive","type":"text","marks":[{"type":"code"}]},{"text":" option to ","type":"text"},{"text":"srun","type":"text","marks":[{"type":"code"}]},{"text":" in order to assure no other users are running jobs on the requested node and it may take a while to get a free node, depending on how many users are running jobs. As in the previous section, the ","type":"text"},{"text":"-c","type":"text","marks":[{"type":"code"}]},{"text":" option option can be used to specify the number of cores, with 16 being the maximum","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"To configure PCT, from the ","type":"text"},{"text":"Home","type":"text","marks":[{"type":"code"}]},{"text":" tab, under ","type":"text"},{"text":"Environment","type":"text","marks":[{"type":"code"}]},{"text":", select the down arrow next to ","type":"text"},{"text":"Parallel","type":"text","marks":[{"type":"code"}]},{"text":" and select ","type":"text"},{"text":"Create and manage a cluster","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This opens up ","type":"text"},{"text":"Cluster Profile Manager","type":"text","marks":[{"type":"code"}]},{"text":". Click ","type":"text"},{"text":"Add cluster profile","type":"text","marks":[{"type":"code"}]},{"text":" and choose ","type":"text"},{"text":"Local (use the cores on your machine)","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click ","type":"text"},{"text":"Set as default","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click ","type":"text"},{"text":"Edit","type":"text","marks":[{"type":"code"}]},{"text":", which is within the ","type":"text"},{"text":"Manage Profile","type":"text","marks":[{"type":"code"}]},{"text":" section.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Add a ","type":"text"},{"text":"Description","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Set ","type":"text"},{"text":"Number of workers","type":"text","marks":[{"type":"code"}]},{"text":" to start on your local machine, with the maximum being 16.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The rest of the fields can be left to their default value, unless you want to change them.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click ","type":"text"},{"text":"Validate,","type":"text","marks":[{"type":"code"}]},{"text":" it can take a minute to complete.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Once validation completes you can use the example for ","type":"text"},{"text":"Use Distributed Arrays to Solve Systems of Linear Equations with Direct Methods","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.mathworks.com/help/releases/R2020b/parallel-computing/Use-Distributed-Arrays-to-Solve-Systems-of-Linear-Equations-with-Direct-Methods.html"}}]},{"text":", you can paste the following into MATLAB's ","type":"text"},{"text":"Editor","type":"text","marks":[{"type":"code"}]},{"text":" and click ","type":"text"},{"text":"Run","type":"text","marks":[{"type":"code"}]},{"text":". A graph labeled ","type":"text"},{"text":"System of Linear Equations with Full Matrices","type":"text","marks":[{"type":"code"}]},{"text":" will pop up. In the bottom left corner under ","type":"text"},{"text":"ondemand (Folder)","type":"text","marks":[{"type":"code"}]},{"text":" you will see ","type":"text"},{"text":"Starting parallel pool","type":"text","marks":[{"type":"code"}]},{"text":" and in the Command Window you will see ","type":"text"},{"text":"Connected to the parallel pool (number of workers: ##)","type":"text","marks":[{"type":"code"}]},{"text":". ## is the number of cores you configured earlier. Assuming you opened MATLAB in the background, you can also run the ","type":"text"},{"text":"top -i","type":"text","marks":[{"type":"code"}]},{"text":" command to see the number of CPUs being used during the calculation.","type":"text"}]},{"type":"codeBlock","content":[{"text":"n = 1e3;\nA = randi(100,n,n);\nADist = distributed(A);\nb = sum(A,2);\nbDist = sum(ADist,2);\nxEx = ones(n,1);\nxDistEx = ones(n,1,'distributed');\nx = A\\b;\nerr = abs(xEx-x);\n\nxDist = ADist\\bDist;\nerrDist = abs(xDistEx-xDist);\n\nfigure\nsubplot(2,1,1)\nsemilogy(err,'o');\ntitle('System of Linear Equations with Full Matrices');\nylabel('Absolute Error');\nxlabel('Element in x');\nylim([10e-17,10e-13])\nsubplot(2,1,2)\nsemilogy(errDist,'o');\ntitle('System of Linear Equations with Distributed Full Matrices');\nylabel('Absolute Error');\nxlabel('Element in x');\nylim([10e-17,10e-13])","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"MATLAB with Parallel Server","type":"text"}]},{"type":"paragraph","content":[{"text":"Running MATLAB via X11 Forwarding","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"MATLAB Parallel Server is now configured on Insomnia for R2022b and R2023a. ","type":"text"},{"text":"X11 Forwarding","type":"text","marks":[{"type":"link","attrs":{"href":"https://goteleport.com/blog/x11-forwarding/"}}]},{"text":" is available and for Apple Mac computers, ","type":"text"},{"text":"XQuartz","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.xquartz.org/"}}]},{"text":" is recommended and for Windows, ","type":"text"},{"text":"MobaXterm","type":"text","marks":[{"type":"link","attrs":{"href":"https://mobaxterm.mobatek.net/download-home-edition.html"}}]},{"text":". The first time you run MATLAB via X11, it can take a few minutes to fully open, especially over WiFi. You can run one simple command to enable the Toolbox:","type":"text"}]},{"type":"paragraph","content":[{"text":">> configCluster","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"You should see:","type":"text"},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Must set AccountName before submitting jobs to Insomnia. E.g.","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":" >> c = parcluster;","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":" >> c.AdditionalProperties.AccountName = 'group-account-name';","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":" >> c.saveProfile","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"Complete. Default cluster profile set to \"Insomnia\".","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Running MATLAB From Your Desktop/Laptop","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"You can now also install MATLAB on your laptop/desktop and download it from ","type":"text"},{"text":"MathWorks Columbia page","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.mathworks.com/academia/tah-portal/columbia-university-650045.html"}}]},{"text":", where students can download it for free. You will need to ","type":"text"},{"text":"download and install a zip file","type":"text","marks":[{"type":"link","attrs":{"href":"https://columbiauniversity.atlassian.net/wiki/download/attachments/62140801/Parallel-Computing-Workshop.zip?version=1&modificationDate=1696436975456&api=v2"}}]},{"text":" which contains all the necessary integration scripts including the license. You will also need to be on the Columbia WiFi or VPN and copy the ","type":"text"},{"text":"network.lic","type":"text","marks":[{"type":"code"}]},{"text":" file into your device's MATLAB directory or running the ","type":"text"},{"text":"userpath","type":"text","marks":[{"type":"code"}]},{"text":" command. On a Mac, you would use Finder, Applications, MATLAB, ctl-click the mouse, Show Package Contents, then licenses. In MATLAB, navigate to the Coumbia-University.Desktop folder. In the Command Window type ","type":"text"},{"text":"configCluster","type":"text","marks":[{"type":"code"}]},{"text":". You will be prompted for Insomnia and Terremoto, select 1, for Insomnia. Enter your UNI (without @columbia.edu). You should see:","type":"text"}]},{"type":"paragraph","content":[{"text":">> c = parcluster;","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":" >> c.AdditionalProperties.AccountName = 'group-account-name';","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":" >> c.saveProfile","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"Complete. Default cluster profile set to \"Insomnia\".","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"paragraph","content":[{"text":"Inside the zip file is a Getting Started tutorial in a Word document. An example interactive pool job follows:","type":"text"}]},{"type":"paragraph","content":[{"text":">> c = parcluster;","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"Submission to the remote cluster requires SSH credentials. You will be prompted for your SSH username and password or identity file (private key). The username and location of the private key will be stored in MATLAB for future sessions. Jobs will now default to the cluster rather than submit to the local machine.","type":"text"}]},{"type":"paragraph","content":[{"text":"Configuring Jobs","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"Prior to submitting the job, we can specify various parameters to pass to our jobs, such as queue, e-mail, walltime, etc. See AdditionalProperties for the complete list. AccountName and MemPerCPU are the only fields that are mandatory.","type":"text"}]},{"type":"paragraph","content":[{"text":">> % Specify the account","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":">> c.AdditionalProperties.AccountName = 'group-account-name';","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":">> % Specify memory to use, per core (default: 4gb)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":">> c.AdditionalProperties.MemPerCPU = '6gb';","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Python and JULIA","type":"text"}]},{"type":"paragraph","content":[{"text":"To use python you need to use:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ module load anaconda\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Here's a simple python program called \"example.py\" – it has just one line:","type":"text"}]},{"type":"codeBlock","content":[{"text":"print (\"Hello, World!\")","type":"text"}]},{"type":"paragraph","content":[{"text":"Save as example.py.","type":"text"}]},{"type":"paragraph","content":[{"text":"To submit it on the Insomnia Cluster, use the submit script \"example.sh\"","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/sh\n#\n# Simple \"Hello World\" submit script for Slurm.\n#\n#SBATCH --account=ACCOUNT # Replace ACCOUNT with your group account name\n#SBATCH --job-name=HelloWorld # The job name.\n#SBATCH -c 1 # The number of cpu cores to use\n#SBATCH -t 0-0:30 # Runtime in D-HH:MM\n#SBATCH --mem-per-cpu=5gb # The memory the job will use per cpu core\n\nmodule load anaconda\n\n#Command to execute Python program\npython example.py\n\n#End of script\n","type":"text"}]},{"type":"paragraph","content":[{"text":"If you use \"ls\" command you should see 2 programs:","type":"text"}]},{"type":"codeBlock","content":[{"text":"example.sh\nexample.py\n","type":"text"}]},{"type":"paragraph","content":[{"text":"To submit it - please use:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ sbatch example.sh\n","type":"text"}]},{"type":"paragraph","content":[{"text":"To check the output use:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ cat slurm-463023.out\nHello, World!\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Similarly, here is the \"julia_example.jl\" with just one line","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ cat julia_example.jl\nprintln(\"hello world\")\n","type":"text"}]},{"type":"paragraph","content":[{"text":"and","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ cat julia_example.sh\n","type":"text"}]},{"type":"paragraph"},{"type":"codeBlock","content":[{"text":"#!/bin/sh\n#\n# Simple \"Hello World\" submit script for Slurm.\n#\n#SBATCH --account=ACCOUNT # Replace ACCOUNT with your group account name \n#SBATCH --job-name=HelloWorld # The job name\n#SBATCH -c 1 # The number of cpu cores to use\n#SBATCH --time=1:00 # The time the job will take to run\n#SBATCH --mem-per-cpu=5gb # The memory the job will use per cpu core\n\nmodule load julia\n\n#Command to execute Python program\njulia julia_example.jl\n\n#End of script\n","type":"text"}]},{"type":"paragraph","content":[{"text":"After you finish creating those two files, if you use \"ls\"command you should see:","type":"text"}]},{"type":"codeBlock","content":[{"text":"julia_example.jl\njulia_example.sh\n","type":"text"}]},{"type":"paragraph","content":[{"text":"To submit it use:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ sbatch julia_example.sh\nSubmitted batch job 463030\n","type":"text"}]},{"type":"paragraph","content":[{"text":"To check the output","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ cat slurm-463030.out\nhello world\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Julia Interactive Session Usage:","type":"text"}]},{"type":"paragraph","content":[{"text":"Step 1 >> start an interactive session (*** replace ACCOUNT with your slurm group account name below):","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ srun --pty -t 0-04:00 -A ACCOUNT /bin/bash\n$ module load julia\n$ julia julia_example.jl\nhello world\n\n$ julia\n_\n_ _ _(_)_ | A fresh approach to technical computing\n() | () (_) | Documentation: http://docs.julialang.org \n_ _ _| |_ __ _ | Type \"?help\" for help.\n| | | | | | |/ _` | |\n| | || | | | (| | | \n_/ |_'|||_'| | Official http://julialang.org/ release\n|__/ | x86_64-pc-linux-gnu\n\njulia>\n\nTo quit Julia use \"CTRL +D\"\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Julia packages can be installed with this command (for example \"DataFrames\" package):","type":"text"}]},{"type":"paragraph"},{"type":"codeBlock","content":[{"text":"julia> using Pkg\njulia> Pkg.add(\"DataFrames\")\n","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Please check this website:","type":"text"},{"type":"hardBreak"},{"text":"https://julialang.org/packages/","type":"text","marks":[{"type":"link","attrs":{"href":"https://julialang.org/packages/"}}]}]},{"type":"paragraph","content":[{"text":"to see the full list of the official packages available.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Snakemake","type":"text"}]},{"type":"paragraph","content":[{"text":"The Snakemake workflow ","type":"text"},{"text":"management","type":"text","marks":[{"type":"code"}]},{"text":" system is a tool to create reproducible and scalable data analyses. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments, without the need to modify the workflow definition.","type":"text"}]},{"type":"paragraph","content":[{"text":"To use it, follow the below steps:","type":"text"}]},{"type":"codeBlock","content":[{"text":"ml anaconda\nconda init\nsource ~/.bashrc\nconda activate snakemake","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Tensorflow","type":"text"}]},{"type":"paragraph","content":[{"text":"Tensorflow computations can use CPUs or GPUs. The default is to use CPUs which are more prevalent, but typically slower than GPUs. ","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Tensorflow ","type":"text"}]},{"type":"paragraph","content":[{"text":"FIRST, start an srun session to a compute node, specifying a GPU if you want to use a GPU.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#414042"}}]}]},{"type":"codeBlock","content":[{"text":"$ srun --pty -t 0-02:00 -A /bin/bash\n\n OR\n$ srun --pty -t 0-02:00 --gres=gpu:1 -A /bin/bash","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"Then use PIP to install Tensorflow. Note that there no longer a separate \"Tensorflow-GPU\" Just install the one version and if you're on a GPU node, you can use the GPU.","type":"text"}]},{"type":"paragraph"},{"type":"codeBlock","content":[{"text":"$ pip install tensorflow","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Test tensorflow ","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"},{"text":"$ python","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"Python 3.7.1 (default, Dec 14 2018, 19:28:38)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":">>> import tensorflow as tf","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":">>> print(tf.__version__)","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":"2.15.0","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"OR","type":"text","marks":[{"type":"strong"}]},{"text":" if on a GPU node - ","type":"text"}]},{"type":"paragraph","content":[{"text":"$ python","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":">>> import tensorflow as tf","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]},{"type":"hardBreak"},{"text":">>> print(\"Num GPUs Available: \", len(tf.config.list_physical_devices('GPU')))","type":"text","marks":[{"type":"subsup","attrs":{"type":"sup"}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Jupyter Lab (formerly Jupyter Notebook)","type":"text"}]},{"type":"paragraph","content":[{"text":"This is one way to set up and run a Jupyter Lab notebook on Insomnia. As your notebook will listen on a port that will be accessible to anyone logged in on a submit node you should first create a password.","type":"text"}]},{"type":"paragraph","content":[{"text":"NOTE: ","type":"text","marks":[{"type":"strong"}]},{"text":"at step 5 you will be editing your jupyter configuration file on Insomnia. You will first need to understand how to use a command line text editor. Advanced users may use \"vi\". Those less advanced should use \"nano\". One tutorial about nano is ","type":"text"},{"text":"at this link","type":"text","marks":[{"type":"link","attrs":{"href":"https://linuxize.com/post/how-to-use-nano-text-editor/"}}]},{"text":", but you can google for others.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Creating a Password","type":"text"}]},{"type":"paragraph","content":[{"text":"1. First, from the initial login node, you will open a shell on a compute node (the below example is for a ","type":"text"},{"text":"CPU","type":"text","marks":[{"type":"underline"}]},{"text":" node)","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ srun --pty -t 0-02:00 -A /bin/bash","type":"text"}]},{"type":"paragraph","content":[{"text":"Please note that the example above specifies time limit of two hours only. That can be set to a much higher value, and in fact the default (i.e. if not specified at all) is as long as 5 days.","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"1B. Alternatively, if you want to do the same thing, but you want to use a ","type":"text"},{"text":"GPU node","type":"text","marks":[{"type":"strong"}]},{"text":" with your notebook, you'll start an srun session to a GPU node like below:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ srun --pty -t 0-02:00 --gres=gpu:1 -A /bin/bash","type":"text"}]},{"type":"paragraph","content":[{"text":"NOTE: ","type":"text","marks":[{"type":"strong"}]},{"text":"On Insomnia, this loads the CUDA module for you automatically.","type":"text"}]},{"type":"paragraph","content":[{"text":"(For more about using GPUs, see the ","type":"text"},{"text":"GPU section of the documentation","type":"text","marks":[{"type":"link","attrs":{"href":"https://columbiauniversity.atlassian.net/wiki/display/rcs/Insomnia+-+Job+Examples#InsomniaJobExamples-GPU(CUDAC/C++)"}}]},{"text":")","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"2. On the compute node, you will next load the anaconda module and initialize the jupyter server. A LOT of installation messages will fly by. That is normal.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ ml anaconda; pip install jupyter","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"3. Get rid of the XDG_RUNTIME_DIR variable and initialize your jupyter environment. ","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ unset XDG_RUNTIME_DIR; jupyter lab --generate-config\n","type":"text"}]},{"type":"paragraph","content":[{"text":"2. Start an ipython session.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ ipython","type":"text"}]},{"type":"paragraph","content":[{"text":"3. Run the password hash generator. You will be prompted for a password, prompted again to verify, and then a hash of that password will be displayed.","type":"text"}]},{"type":"paragraph","content":[{"text":"DO NOT USE YOUR UNI PASSWORD (or something from your bank, social media, etc.) Create a new one just for this.","type":"text"}]},{"type":"codeBlock","content":[{"text":"In [1]: from jupyter_server.auth import passwd; passwd()\nEnter password:\nVerify password:\nOut[1]: 'sha1:60bdb1:306fe0101ca73be2429edbab0935c545'\nIn [2]: exit","type":"text"}]},{"type":"paragraph","content":[{"text":"4. Copy the hash line and then type \"exit\" to exit iPython. You may want to open a local Notepad or Textedit window on your computer to paste that line in so it won't get lost. We'll need it in a moment.","type":"text"}]},{"type":"paragraph","content":[{"text":"5. Use a text editor to place the line at the right spot in your ~/.jupyter/jupyter_lab_config.py file (i.e. \"nano ~/.jupyter/jupyter_lab_config.py\")","type":"text"}]},{"type":"paragraph","content":[{"text":"(","type":"text"},{"text":"Important","type":"text","marks":[{"type":"strong"}]},{"text":": the following line in the file is commented out by default so please uncomment it first)","type":"text"}]},{"type":"codeBlock","content":[{"text":"CHANGE # c.ServerApp.password = '' INTO c.ServerApp.password = 'sha1:60bdb1:306fe0101ca73be2429edbab0935c545'\n\n(Remember, your hash will look different. Do not use this example one.)","type":"text"}]},{"type":"paragraph","content":[{"text":"Setting the password will prevent other users from having access to your notebook and potentially causing confusion.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Running the Jupyter Notebook","type":"text"}]},{"type":"paragraph","content":[{"text":"6. Look up the IP of the node your interactive job is running on.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ hostname -i\n10.197.80.51 ","type":"text"}]},{"type":"paragraph","content":[{"text":"7. Start the Jupyter Lab notebook, specifying the node IP.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ jupyter lab --no-browser --ip=10.197.80.51 ","type":"text"}]},{"type":"paragraph","content":[{"text":"8. Look for the following 2 lines in the startup output to get the port number.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#172b4d"}}]}]},{"type":"codeBlock","content":[{"text":"Jupyter Server 2.15.0 is running at:\nhttp://10.197.80.51:8888/lab","type":"text"}]},{"type":"paragraph","content":[{"text":"9. From your local system, open a second connection window to Insomnia that forwards a local port to the remote node and port. Replace UNI below with your uni.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ ssh -L 8080:10.197.80.51:8888 UNI@som.rcs.columbia.edu (This is not for Windows users. Windows users, see step 9B).\n","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"9B. Windows users generally are using PuTTY and not a native command line, so step 9 instructions, which use Port Forwarding, may be particularly hard to replicate. To accomplish Step 9 while using PuTTY, you should do this -","type":"text"}]},{"type":"paragraph","content":[{"text":"I. Open PuTTY.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"II. In the \"Session\" category on the left side, enter the hostname or IP address of the remote server in the \"Host Name (or IP address)\" field. (In this case - som.rcs.columbia.edu).","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"III. Make sure the connection type is set to SSH.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"IV. In the \"Connection\" category, expand the \"SSH\" tab and select \"Tunnels\".","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"V. In the \"Source port\" field, enter 8080.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"VI. In the \"Destination\" field, enter ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"text":"10.197.80.51:8888","type":"text","marks":[{"type":"code"}]},{"text":"(Remember, this is only an example IP. the one you use will be different)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"VII. Make sure the \"Local\" radio button is selected.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"VIII. Click the \"Add\" button to add the port forwarding rule to the list.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"IX. Now, return to the \"Session\" category on the left side.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"X. Optionally, enter a name for this configuration in the \"Saved Sessions\" field, then","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"XI. Click \"Save\" to save these settings for future use.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]},{"type":"hardBreak"},{"text":"XII. Click \"Open\" to start the SSH connection with the port forwarding configured.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#455464"}}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"10. LAST STEP - Open a browser session on your desktop and enter the URL 'localhost:8080' (i.e. the string within the single quotes) into its search field. ","type":"text"}]},{"type":"paragraph","content":[{"text":" IF YOU DO NOT HAVE A PREVIOUS NOTEBOOK, you will see a main window in front of you with several options: Notebook, Console, and Other. Choose the Python kernel under the Notebook button on the top. This will open a brand new Jupyter Lab notebook for you.","type":"text"}]},{"type":"paragraph","content":[{"text":" IF YOU DO HAVE AN EXISTING NOTEBOOK FROM BEFORE, that notebook will likely open directly","type":"text"}]},{"type":"paragraph","content":[{"text":" NOTE: if the browser window does not work the first time, you should try quitting your browser entirely and restarting with a new window, opening a new Private or Incognito window, or also trying another browser. Some popular browsers are Safari, Chrome, and Firefox. Sometimes one may work where another does not.","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Getting Python libraries JAX and JAXLIB to work with GPUs","type":"text"}]},{"type":"paragraph","content":[{"text":"JAX","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/google/jax"}}]},{"text":", a Just-In-Time (JIT) compiler focused on harnessing the maximum number of FLOPs to generate optimized code while using the simplicity of pure Python. It is frequently updated with ","type":"text"},{"text":"strict version requirements","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/google/jax#instructions"}}]},{"text":" for minimum versions of CUDA and CUDNN. The following combination of modules and libraries will work and remember to request a GPU node. Note to use","type":"text"}]},{"type":"codeBlock","content":[{"text":"module load cuda\nconda deactivate (If you have the conda environment initialized from previous work, it will interfere)\npython\n\nPython 3.9.18 (main, Jan 4 2024, 00:00:00) \n[GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n\n>>> from jax.lib import xla_bridge\n>>> print(xla_bridge.get_backend().platform)\ngpu","type":"text"}]},{"type":"paragraph","content":[{"text":"If you'd like to install the correct versions for your user run:","type":"text"}]},{"type":"paragraph","content":[{"text":"pip install -U jax==0.4.7 -f ","type":"text","marks":[{"type":"code"}]},{"text":"https://storage.googleapis.com/jax-releases/jax_cuda_releases.html","type":"text","marks":[{"type":"link","attrs":{"href":"https://storage.googleapis.com/jax-releases/jax_cuda_releases.html"}}]},{"text":" ","type":"text","marks":[{"type":"code"}]},{"text":"https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.4.7+cuda11.cudnn86-cp39-cp39-manylinux2014_x86_64.whl","type":"text","marks":[{"type":"link","attrs":{"href":"https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.4.7+cuda11.cudnn86-cp39-cp39-manylinux2014_x86_64.whlpython"}}]}]}],"version":1}

Browser not supported