| Table of Contents | ||
|---|---|---|
|
...
This program will print out "Hello World!" when run on a gpu server or print "Hello Hello" when no gpu module is found.
Singularity
Singularity is a software tool that brings Docker-like containers and reproducibility to scientific computing and HPC. Singularity has Docker container support and enables users to easily run different flavors of Linux with different software stacks. These containers provide a single universal on-ramp from the laptop, to HPC, to cloud.
Users can run Singularity containers just as they run any other program on our HPC clusters. Example usage of Singularity is listed below. For additional details on how to use Singularity, please contact us or refer to the Singularity User Guide.
Downloading Pre-Built Containers
Singularity makes it easy to quickly deploy and use software stacks or new versions of software. Since Singularity has Docker support, users can simply pull existing Docker images from Docker Hub or download docker images directly from software repositories that increasingly support the Docker format. Singularity Container Library also provides a number of additional containers.
You can use the pull command to download pre-built images from an external resource into your current working directory. The docker:// uri reference can be used to pull Docker images. Pulled Docker images will be automatically converted to the Singularity container format.
...
Here's an example of pulling the latest stable release of the Tensorflow Docker image and running it with Singularity. (Note: these pre-built versions may not be optimized for use with our CPUs.)
...
Singularity - Interactive Shell
The shell command allows you to spawn a new shell within your container and interact with it as though it were a small virtual machine:
...
| Code Block |
|---|
Singularity tensorflow.simg:~> python >>> import tensorflow as tf >>> print(tf.__version__) 1.13.1 >>> exit() |
When done, you may exit the Singularity interactive shell with the "exit" command.
Singularity tensorflow.simg:~> exit
Singularity: Executing Commands
The exec command allows you to execute a custom command within a container by specifying the image file. This is the way to invoke commands in your job submission script.
...
Singularity: Running a Batch Job
Below is an example of job submission script named submit.sh that runs Singularity. Note that you may need to specify the full path to the Singularity image you wish to run.
| Code Block |
|---|
#!/bin/bash # Singularity example submit script for Slurm. # # Replace <ACCOUNT> with your account name before submitting. # #SBATCH -A <ACCOUNT> # Set Account name #SBATCH --job-name=tensorflow # The job name #SBATCH -c 1 # Number of cores #SBATCH -t 0-0:30 # Runtime in D-HH:MM #SBATCH --mem-per-cpu=4gb # Memory per cpu core module load singularity singularity exec tensorflow.simg python -c 'import tensorflow as tf; print(tf.__version__)' |
Then submit the job to the scheduler. This example prints out the tensorflow version.
$ sbatch submit.sh
For additional details on how to use Singularity, please contact us or refer to the Singularity User Guide.
swak4Foam in a Singularity container
swak4Foam (SWiss Army Knife for Foam) can be run inside a container. Using this Docker container as inspiration, here is a sample tutorial.
| Code Block |
|---|
module load singularity
singularity pull docker://hfdresearch/swak4foamandpyfoam:latest-v7.0
singularity shell swak4foamandpyfoam_latest-v7.0.sif |
From the pulsedPitzDaily tutorial about halfway down the page, change to use /local as it has more space than /tmp:
| Code Block |
|---|
source /opt/openfoam7/etc/bashrc
cd /local
cp -r /opt/swak4Foam/Examples/groovyBC/pulsedPitzDaily .
cd pulsedPitzDaily
pyFoamPrepareCase.py . |
You should see the following output:
| Code Block |
|---|
Looking for template values .
Used values
Name - Value
----------------------------------------
caseName - "pulsedPitzDaily"
casePath - "/local/pulsedPitzDaily"
foamFork - openfoam
foamVersion - 7
numberOfProcessors - 1
No script ./derivedParameters.py for derived values
Clearing .
Clearing /local/pulsedPitzDaily/PyFoam.blockMesh.logfile
Clearing /local/pulsedPitzDaily/PyFoamPrepareCaseParameters
Writing parameters to ./PyFoamPrepareCaseParameters
Writing report to ./PyFoamPrepareCaseParameters.rst
Found 0.org . Clearing 0
Looking for templates with extension .template in /local/pulsedPitzDaily
Looking for templates with extension .template in /local/pulsedPitzDaily/0.org
Looking for templates with extension .template in /local/pulsedPitzDaily/constant
Looking for templates with extension .template in /local/pulsedPitzDaily/constant/polyMesh
Found template for /local/pulsedPitzDaily/constant/LESProperties
Found template for /local/pulsedPitzDaily/constant/turbulenceProperties
Looking for templates with extension .template in /local/pulsedPitzDaily/system
No script for mesh creation found. Looking for 'blockMeshDict'
/local/pulsedPitzDaily/constant/polyMesh/blockMeshDict found. Executing 'blockMesh'
/*---------------------------------------------------------------------------*\
========= |
\\ / F ield | OpenFOAM: The Open Source CFD Toolbox
\\ / O peration | Website: https://openfoam.org
\\ / A nd | Version: 7
\\/ M anipulation |
\*---------------------------------------------------------------------------*/
Build : 7-1ff648926f77
Exec : blockMesh -case /local/pulsedPitzDaily
Date : May 03 2023
Time : 16:23:59
PID : 2082964
I/O : uncollated
Case : /local/pulsedPitzDaily
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time
Not deleting polyMesh directory
"/local/pulsedPitzDaily/constant/polyMesh"
because it contains blockMeshDict
Creating block mesh from
"/local/pulsedPitzDaily/constant/polyMesh/blockMeshDict"
Creating block edges
No non-planar block faces defined
Creating topology blocks
Creating topology patches
Reading patches section
Creating block mesh topology
Reading physicalType from existing boundary file
Default patch type set to empty
Check topology
Basic statistics
Number of internal faces : 18
Number of boundary faces : 42
Number of defined boundary faces : 42
Number of undefined boundary faces : 0
Checking patch -> block consistency
Creating block offsets
Creating merge list .
Creating polyMesh from blockMesh
Creating patches
Creating cells
Creating points with scale 0.001
Block 0 cell size :
i : 0.00158284 .. 0.000791418
j : 0.000313389 .. 0.000564101
k : 0.001
Block 1 cell size :
i : 0.00158284 .. 0.000791418
j : 0.000440611 .. 0.00176244
k : 0.001
Block 2 cell size :
i : 0.00158284 .. 0.000791418
j : 0.00178262 .. 0.000445655
k : 0.001
Block 3 cell size :
i : 0.000528387 .. 0.00211355
j : 0.00113333 .. 0.00113333 0.00113283 .. 0.00113283 0.00113333 .. 0.00113333 0.00113283 .. 0.00113283
k : 0.001
Block 4 cell size :
i : 0.000528464 .. 0.00211385 0.000528454 .. 0.00211383 0.000528464 .. 0.00211385 0.000528454 .. 0.00211383
j : 0.000766355 .. 0.000383178 0.000766938 .. 0.000384514 0.000766355 .. 0.000383178 0.000766938 .. 0.000384514
k : 0.001
Block 5 cell size :
i : 0.000528387 .. 0.00211355
j : 0.000313389 .. 0.000564101 0.000314853 .. 0.00056517 0.000313389 .. 0.000564101 0.000314853 .. 0.00056517
k : 0.001
Block 6 cell size :
i : 0.000528464 .. 0.00211385 0.000528492 .. 0.00211397 0.000528464 .. 0.00211385 0.000528492 .. 0.00211397
j : 0.000440611 .. 0.00176244 0.000442137 .. 0.00176067 0.000440611 .. 0.00176244 0.000442137 .. 0.00176067
k : 0.001
Block 7 cell size :
i : 0.000528502 .. 0.00211401 0.000528472 .. 0.00211389 0.000528502 .. 0.00211401 0.000528472 .. 0.00211389
j : 0.00178262 .. 0.000445655 0.00178107 .. 0.000445268 0.00178262 .. 0.000445655 0.00178107 .. 0.000445268
k : 0.001
Block 8 cell size :
i : 0.0020578 .. 0.00514451 0.00205689 .. 0.00514223 0.0020578 .. 0.00514451 0.00205689 .. 0.00514223
j : 0.000938889 0.000929955 .. 0.000929955 0.000938889 0.000929955 .. 0.000929955
k : 0.001
Block 9 cell size :
i : 0.00204731 .. 0.00511826 0.00204716 .. 0.0051179 0.00204731 .. 0.00511826 0.00204716 .. 0.0051179
j : 0.000944444 .. 0.000944444 0.000938489 .. 0.000938489 0.000944444 .. 0.000944444 0.000938489 .. 0.000938489
k : 0.001
Block 10 cell size :
i : 0.0020466 .. 0.00511651
j : 0.000928571 .. 0.000928571 0.00092161 .. 0.00092161 0.000928571 .. 0.000928571 0.00092161 .. 0.00092161
k : 0.001
Block 11 cell size :
i : 0.00204718 .. 0.00511796 0.00204744 .. 0.0051186 0.00204718 .. 0.00511796 0.00204744 .. 0.0051186
j : 0.00105 .. 0.00105 0.00104025 .. 0.00104025 0.00105 .. 0.00105 0.00104025 .. 0.00104025
k : 0.001
Block 12 cell size :
i : 0.00205182 .. 0.00512954 0.00205252 .. 0.00513131 0.00205182 .. 0.00512954 0.00205252 .. 0.00513131
j : 0.00117906 .. 0.000294764 0.00116948 .. 0.00029237 0.00117906 .. 0.000294764 0.00116948 .. 0.00029237
k : 0.001
Adding cell zones
0 center
Writing polyMesh
----------------
Mesh Information
----------------
boundingBox: (-0.0206 -0.0254 -0.0005) (0.29 0.0254 0.0005)
nPoints: 25012
nCells: 12225
nFaces: 49180
nInternalFaces: 24170
----------------
Patches
----------------
patch 0 (start: 24170 size: 30) name: inlet
patch 1 (start: 24200 size: 57) name: outlet
patch 2 (start: 24257 size: 223) name: upperWall
patch 3 (start: 24480 size: 250) name: lowerWall
patch 4 (start: 24730 size: 24450) name: frontAndBack
End
No mesh decomposition necessary
Looking for originals in /local/pulsedPitzDaily
Looking for originals in /local/pulsedPitzDaily/constant
Looking for originals in /local/pulsedPitzDaily/constant/polyMesh
Looking for originals in /local/pulsedPitzDaily/system
Copying /local/pulsedPitzDaily/0.org to /local/pulsedPitzDaily/0
No field decomposition necessary
Looking for templates with extension .postTemplate in /local/pulsedPitzDaily
Looking for templates with extension .postTemplate in /local/pulsedPitzDaily/0.org
Looking for templates with extension .postTemplate in /local/pulsedPitzDaily/constant
Looking for templates with extension .postTemplate in /local/pulsedPitzDaily/constant/polyMesh
Looking for templates with extension .postTemplate in /local/pulsedPitzDaily/system
Looking for templates with extension .postTemplate in /local/pulsedPitzDaily/0
No script for case-setup found. Nothing done
No case decomposition necessary
Looking for templates with extension .finalTemplate in /local/pulsedPitzDaily
Looking for templates with extension .finalTemplate in /local/pulsedPitzDaily/0.org
Looking for templates with extension .finalTemplate in /local/pulsedPitzDaily/constant
Looking for templates with extension .finalTemplate in /local/pulsedPitzDaily/constant/polyMesh
Looking for templates with extension .finalTemplate in /local/pulsedPitzDaily/system
Looking for templates with extension .finalTemplate in /local/pulsedPitzDaily/0
Clearing templates
Looking for extension .template in /local/pulsedPitzDaily/0
Looking for extension .postTemplate in /local/pulsedPitzDaily/0
Looking for extension .finalTemplate in /local/pulsedPitzDaily/0
Case setup finished |
Example of R run
For this example, the R code below is used to generate a graph ''Rplot.pdf'' of a discrete Delta-hedging of a call. It hedges along a path and repeats over many paths. There are two R files required:
...
| Code Block |
|---|
$ srun --pty -t 0-02:00:00 --gres=gpu:1 -A <group_name> /bin/bash |
Then load the singularity environment module and run the tensorflow container, which was built from the Tensorflow docker image. You can start an interactive singularity shell and specify the --nv flag which instructs singularity to use the Nvidia GPU driver.
| Code Block |
|---|
$ module load singularity
$ singularity shell --nv /moto/opt/singularity/tensorflow-1.13-gpu-py3-moto.simg
Singularity tensorflow-1.13-gpu-py3-moto.simg:~> python
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux
>>> import tensorflow as tf
>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
..
>>> exit() |
You may type "exit" to exit when you're done with the Singularity shell.
Singularity tensorflow-1.13-gpu-py3-moto.simg:~> exit
Below is an example of job submission script named submit.sh that runs Tensorflow with GPU support using Singularity.
| Code Block |
|---|
#!/bin/bash # Tensorflow with GPU support example submit script for Slurm. # # Replace <ACCOUNT> with your account name before submitting. # #SBATCH -A <ACCOUNT> # Set Account name #SBATCH --job-name=tensorflow # The job name #SBATCH -c 1 # Number of cores #SBATCH -t 0-0:30 # Runtime in D-HH:MM #SBATCH --gres=gpu:1 # Request a gpu module module load singularity singularity exec --nv /moto/opt/singularity/tensorflow-1.13-gpu-py3-moto.simg python -c 'import tensorflow as tf; print(tf.__version__)' |
Then submit the job to the scheduler. This example prints out the tensorflow version.
$ sbatch submit.sh
For additional details on how to use Singularity, please contact us, see our Singularity documentation, or refer to the Singularity User Guide.
Another option:
Please note that you should not work on our head node.
...