DeepLabCut

DeepLabCut is not installed globally on Axon. This is in part due to the rapid release cycle of DeepLabCut (any version installed globally would quickly become stale), but also because the ability to install DeepLabCut locally in a conda environment without requiring administrator privileges encourages rapid iteration and experimentation (i.e., Zuckerman research computing doesn't want the limiting factor for you to be time spent waiting for us to install software amidst our competing priorities). We thus currently only support DeepLabCut when installed within a local conda environment.

That said, there are some particular issues that must be kept in mind when installing DeepLabCut within a conda environment on Axon. Below we provide installation steps that are known to work on Axon, as well as common issues encountered.

Deep Lab Cut can be tricky to install, primarily because it depends on Tensorflow, which is itself sometimes tricky to install.

First we suggest installing Tensorflow as outlined here then you will be able to install DLC with pip:

conda install pip ipython jupyter ffmpeg notebook=6.5.6 nb_conda -c conda-forge
pip install deeplabcut

Below are out of date instructions which may be of use for those requiring older versions.

Installing and Using DeepLabCut in Headless Mode

By default, DeepLabCut assumes the presence of a desktop environment. Because Axon is designed for batch processing, there is no graphical user interface installed on any of its nodes. For DeepLabCut to run properly in such a batch environment, special measures must be taking when installing and using DeepLabCut. The following instructions are adapted from the official instructions here, and have been tested as of 4/2/2020 (with a successful test defined as DeepLabCut importing without error). They install version 2.1.6.4 of DeepLabCut, but may be adapted for newer versions.

Activate the Anaconda and CUDA modules on Axon.

ml load anaconda3-2019.03
ml load cuda/10.0.130

Create a new conda environment with Tensorflow and other DeepLabCut dependencies installed (you can replace DLC in the syntax below with another environment name if necessary or desirable).
```
conda create -n DLC python=3.7 numpy=1.16.4 tensorflow-gpu=1.13.1 pip cudnn=7 wxpython jupyter nb_conda Shapely
```
Install DeepLabCut and additional dependencies using pip:
```
conda activate DLC
pip install moviepy==1.0.1
pip install deeplabcut==2.1.6.4
conda install -c conda-forge opencv
```
As of 5/18/20, there is a new headless package for DeepLabCut (see here). This can be installed via the following command:
pip install deeplabcutcore
This package has not yet been tested by Zuckerman Research Computing, but may allow you to use versions of DeepLabCut higher than 2.1.6.4 in a sustainable way.

Once you have finished installing DeepLabCut, you must execute the following command within your submission script for a batch job or within your terminal for an interactive session before attempting to import DeepLabCut in Python:

export DLClight=True

If you do not do this, you will receive an error involving a wx module(described in further detail below).

Common Error Messages

ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

The following error can occur if you are trying to import deeplabcut on a node that does not have a GPU installed (such as the login node, axon.rc.zi.columbia.edu):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/__init__.py", line 43, in <module>
    from deeplabcut.refine_training_dataset import extract_outlier_frames, merge_datasets
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/refine_training_dataset/__init__.py", line 20, in <module>
    from deeplabcut.refine_training_dataset.outlier_frames import *
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/refine_training_dataset/outlier_frames.py", line 17, in <module>
    from deeplabcut.utils import auxiliaryfunctions, visualization
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/utils/__init__.py", line 1, in <module>
    from deeplabcut.utils.make_labeled_video import *
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/utils/make_labeled_video.py", line 38, in <module>
    from deeplabcut.pose_estimation_tensorflow.config import load_config
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/pose_estimation_tensorflow/__init__.py", line 13, in <module>
    from deeplabcut.pose_estimation_tensorflow.nnet import *
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/pose_estimation_tensorflow/nnet/__init__.py", line 14, in <module>
    from deeplabcut.pose_estimation_tensorflow.nnet.losses import *
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/pose_estimation_tensorflow/nnet/losses.py", line 5, in <module>
    import tensorflow as tf
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

When imported on a node with GPUs, this error should not occur.

ImportError: libSM.so.6

When importing DeepLabCut, you may sometimes receive an error that appears as follows:

Python 3.7.7 (default, Mar 26 2020, 15:48:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deeplabcut
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/__init__.py", line 43, in <module>
    from deeplabcut.refine_training_dataset import extract_outlier_frames, merge_datasets
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/refine_training_dataset/__init__.py", line 20, in <module>
    from deeplabcut.refine_training_dataset.outlier_frames import *
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/refine_training_dataset/outlier_frames.py", line 17, in <module>
    from deeplabcut.utils import auxiliaryfunctions, visualization
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/utils/__init__.py", line 1, in <module>
    from deeplabcut.utils.make_labeled_video import *
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/utils/make_labeled_video.py", line 37, in <module>
    from deeplabcut.utils import auxiliaryfunctions
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/deeplabcut/utils/auxiliaryfunctions.py", line 18, in <module>
    import cv2
  File "/home/jsp2205/.conda/envs/DLC/lib/python3.7/site-packages/cv2/__init__.py", line 3, in <module>
    from .cv2 import *
ImportError: libSM.so.6: cannot open shared object file: No such file or directory

libSM.so.6 is a system library used for graphical user interfaces (in particular, it is for the X Session manager- hence, the SM abbreviation). You may receive this error because, as mentioned earlier, Axon is configured to be run as a batch processing system with a text-only interface. This error in particular is caused by the OpenCV package for Python, as indicated by the following line in the stack trace:

    from .cv2 import *

Some variants of the OpenCV package assume the presence a graphical interface, although other variants do not. In particular, the opencv-python package, which DeepLabCut installs automatically as of version 2.1.6.4, assumes that a graphical desktop is installed. Another variant (opencv-python-headless), does not make assumptions about the presence or absence of a graphical user interface and can be used instead to remove this error, but does not support all video codecs out there. Still better, the conda-forge variant package uses a version of ffmpeg that supports more ways of encoding video. If you ever receive an error from OpenCV that implicates a codec, chances are that the OpenCV package that you're using has more limited support.

To prevent DeepLabCut from trying to use a version of OpenCV that assumes a graphical interface, you must overwrite the version of OpenCV that DeepLabCut installs when you run pip install deeplabcut by running another installation command for OpenCV afterwards. Because the conda-forge OpenCV package supports more video codecs, we recommend using the following install command (taken from the installation instructions above) to overwrite the incompatible copy of OpenCV that comes with DeepLabCut:

conda install -c conda-forge opencv

For further reference, please see the following links

ModuleNotFoundError: No module named 'wx'

When using DeepLabCut on a system without a graphical interface, you must tell DeepLabCut to run in light mode (which toggles off the labeling, relabeling and a standalone GUI). You can accomplish this by running export DLClight=True before importing DeepLabCut. If you do not do this, DeepLabCut will attempt to load wxPython, which is a software library used by the DeepLabCut graphical interface to make buttons, scrollbars and windows. If light mode is properly enabled, importing DeepLabCut should look something like the following:

Python 3.7.7 (default, Mar 26 2020, 15:48:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import deeplabcut
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)

For further reference, please see the following links: