/
Virtual Environments

Virtual Environments

Why use virtual environments with SLURM?

Because:

  • Login nodes and compute nodes may have different software versions.

  • You may need specific packages that are not cluster-wide.

  • Reproducibility — your job always runs with the exact same dependency set.

  • Prevent conflicts with other users' packages.

  • Users often need isolated Python packages for their jobs.

  • They create virtual environments in their home or project directory.

  • Then they activate the environment inside a SLURM job script so the job runs with the correct software stack.

 Instructions to create a Conda virtual environment

Here is an example to create a Conda environment using conda create command:

  1. You can create virtual environment in your /users or /groups directory.

  2. Enter the command to create a new virtual environment:

    module load conda python -v conda create --name myenv python=3.12.6 ; where myenv is the env_name.

  3. Activate your virtual environment by running below command:

    conda activate myenv

  4. Once the environment is activated, you can install any Python packages you need using conda or pip. For example, to install numpy package using conda, you can run:

    conda install numpy or pip install numpy

  5. When you’re done working in the environment, you can deactivate it by running the following command:

    conda deactivate

 How to use conda env in slurm script

To use a virtual environment created with conda in a Slurm script, you need to activate the environment before running your Python script:

Here is how you do that:

Replace /path/to/venv with the path to your virtual environment directory, and myscript.py with the name of your Python script.

#!/bin/bash #SBATCH --job-name=myjob #SBATCH --output=myjob.out #SBATCH --error=myjob.err #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --time=1:00:00 #SBATCH --partition=your_partition # Load any necessary modules or dependencies module load conda module load some_module # Activate the virtual environment conda activate /path/to/env # Run your Python script python myscript.py # Deactivate the virtual environment conda deactivate