The following document has additional useful information, make sure to check it out: Cluster Q&A. If you are a part of the SNAP group, GPU usage information is available on the #gpu channel.

• The following document has additional useful information, make sure to check it out: Cluster Q&A
• Thou shalt not run CPU-intensive tasks on the GPU servers, because saturating the CPUs will slow down also the processes running on the GPUs. We have plenty of other servers for that.
• Thou shalt not forget to check the output of “nvidia-smi” after you launch your process, and after you interrupt it. In this way, you'll find out if you are over-using the memory (therefore risking an OOM), and if you released correctly all the resources after you interrupted your process. To err on the side of caution, a simple `ps uxw` will also show you if you have any unexpected processes still allocating resources.

To not allocate the memory on all GPUs but only on the GPU that you are going to use you must make sure that you set CUDA_VISIBLE_DEVICES options to the GPU/GPUs you are actually going to use for your jobs.

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0"

export CUDA_VISIBLE_DEVICES=0

export CUDA_VISIBLE_DEVICES=0,1,2

Please also read this documentation before using GPUs [1] [2].

To see a more detailed GPU usage run the following command:

csid@turing1:~$ nvidia-htop.py

Almost every GPU server has Cuda 10.0 and Cuda 9.0 installed (not yet done on hyperion1 & 3).

To switch between versions you just need to set the environment in the right way. The default Cuda version in use is Cuda 10.0.

source cuda9.0

export PATH=/usr/local/cuda-9.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64:$LD_LIBRARY_PATH

source cuda10.0

export PATH=/usr/local/cuda-10.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64:$LD_LIBRARY_PATH

There are also Cuda 10.1 and 10.2 available.

source cuda10.1
#OR
source cuda10.2

nvcc -V