A growing subset of the SNAP cluster is managed through SLURM. SLURM allocates resources for the jobs you want to run. A key advantage for everyone is that you can use more resources when nodes are otherwise inactive.

SSH into the head node:

ssh <csid>@ilc.stanford.edu

Students should submit jobs to the il partition only. Use QoS (Quality of Service) to choose the kind of job you want:

The partition is still il for all of these. For example, an interactive job uses –partition=il –qos=il-interactive, not an il-interactive partition.

Schedule an interactive shell for an amount of time on a node:

roed@ilc:~$ srun --partition=il --qos=il-interactive --account=infolab --mem=100G -c 12 --gres=gpu:1 --time=4:00:00 --pty bash

This immediately drops you into a shell on the node you were allocated. If you need a specific node, you can pass -w <node-name>. If you need a specific GPU type, request it through –gres:

# One A100 GPU
roed@ilc:~$ srun --partition=il --qos=il-interactive --account=infolab --gres=gpu:a100:1 --time=4:00:00 --pty bash

# One RTX 8000 GPU
roed@ilc:~$ srun --partition=il --qos=il-interactive --account=infolab --gres=gpu:rtx8000:1 --time=4:00:00 --pty bash

# One RTX 2080 Ti GPU
roed@ilc:~$ srun --partition=il --qos=il-interactive --account=infolab --gres=gpu:2080ti:1 --time=4:00:00 --pty bash

You can also request by node feature:

# A100 nodes have the ampere feature.
roed@ilc:~$ srun --partition=il --qos=il-interactive --account=infolab --constraint=ampere --gres=gpu:1 --time=4:00:00 --pty bash

# RTX 8000 and RTX 2080 Ti nodes have the turing feature.
roed@ilc:~$ srun --partition=il --qos=il-interactive --account=infolab --constraint=turing --gres=gpu:1 --time=4:00:00 --pty bash

Once your job starts, check the GPU from the allocated node:

roed@turing3:~$ nvidia-smi

You can SSH into any node where you have a running job:

roed@ilc:~$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             10016        il     bash     roed  R       0:11      1 turing3
roed@ilc:~$ ssh turing3

If you want your command to run without keeping a terminal open, use sbatch. A normal batch job should use the il QoS, which is also the default QoS for the infolab account:

#!/bin/bash
#SBATCH --partition=il
#SBATCH --qos=il
#SBATCH --account=infolab
#SBATCH --time=1-00:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --gres=gpu:1

python train.py

Submit it with:

roed@ilc:~$ sbatch train.sbatch

For low-priority work that can wait or be requeued, use –qos=il-lo:

#!/bin/bash
#SBATCH --partition=il
#SBATCH --qos=il-lo
#SBATCH --account=infolab
#SBATCH --time=21-00:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --gres=gpu:1

python train.py

Low-priority jobs are useful for using idle resources, but they should be able to tolerate being stopped and requeued. Use checkpoints for long-running training jobs.

You can inspect existing jobs using squeue:

roed@ilc:~$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             10016        il     bash     roed  R       6:28      1 turing3

Show only your jobs:

roed@ilc:~$ squeue -u $USER

Kill one of your jobs with scancel <jobid>. The job may briefly show state CG for Completing or Canceling:

roed@ilc:~$ scancel 10016
roed@ilc:~$ squeue -u $USER

Get more information about a job with scontrol show job <jobid>:

roed@ilc:~$ scontrol show job 10016
JobId=10016 JobName=bash
   UserId=roed(23269) GroupId=users(100) MCS_label=N/A
   JobState=RUNNING Reason=None
   Account=infolab QOS=il-interactive
   Partition=il
   NodeList=turing3
   NumNodes=1 NumCPUs=12 NumTasks=1 CPUs/Task=12
   ReqTRES=cpu=12,mem=100G,node=1,billing=12,gres/gpu=1
   AllocTRES=cpu=12,mem=100G,node=1,billing=12,gres/gpu=1,gres/gpu:2080ti=1

Partitions are groups of nodes with shared access rules. Students should use the il partition. You may see dev partitions in scheduler output, but those are not for normal student jobs.

List the available partitions:

roed@ilc:~$ sinfo -a -o "%P|%a|%l|%D|%T|%N"
PARTITION|AVAIL|TIMELIMIT|NODES|STATE|NODELIST
dev|up|7-00:00:00|1|idle|ilcx
dev-interactive|up|3-00:00:00|1|idle|ilcx
il*|up|21-00:00:00|4|mixed-|ampere[1,7-9]
il*|up|21-00:00:00|1|mixed|ampere4
il*|up|21-00:00:00|1|allocated|ampere6
il*|up|21-00:00:00|2|idle|hyperturing2,turing3

The * after il means it is the default partition.

Look at the current il partition details with scontrol show partition il:

roed@ilc:~$ scontrol show partition il
PartitionName=il
   AllowGroups=ALL AllowAccounts=infolab AllowQos=il,il-lo,il-interactive
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=12:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=300 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=21-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   Nodes=ampere[1,4,6-9],hyperturing2,turing3
   PriorityJobFactor=1 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=REQUEUE
   State=UP TotalCPUs=1100 TotalNodes=8 SelectTypeParameters=NONE
   JobDefaults=DefMemPerGPU=240000
   DefMemPerCPU=4096 MaxMemPerCPU=10700
   TRES=cpu=1100,mem=15830000M,node=8,billing=1100,gres/gpu=68,gres/gpu:2080ti=10,gres/gpu:a100=48,gres/gpu:rtx8000=10

Current il nodes:

You can check live node state and allocation with sinfo:

roed@ilc:~$ sinfo -N -o "%N|%P|%t|%c|%m|%G|%f"
NODELIST|PARTITION|STATE|CPUS|MEMORY|GRES|AVAIL_FEATURES
ampere1|il*|mix-|128|2050000|gpu:a100:8|80G,ampere
ampere4|il*|mix|128|2050000|gpu:a100:8|80G,ampere
ampere6|il*|alloc|128|2050000|gpu:a100:8|80G,ampere
ampere7|il*|mix-|128|2050000|gpu:a100:8|80G,ampere
ampere8|il*|mix-|128|2050000|gpu:a100:8|80G,ampere
ampere9|il*|mix-|128|2050000|gpu:a100:8|80G,ampere
hyperturing2|il*|idle|252|2050000|gpu:rtx8000:10|48G,turing
turing3|il*|idle|80|1480000|gpu:2080ti:10|12G,turing

You can inspect a specific node with scontrol show node <node-name>:

roed@ilc:~$ scontrol show node turing3
NodeName=turing3 Arch=x86_64 CoresPerSocket=20
   CPUAlloc=0 CPUEfctv=80 CPUTot=80 CPULoad=0.05
   AvailableFeatures=12G,turing
   ActiveFeatures=12G,turing
   Gres=gpu:2080ti:10
   RealMemory=1480000 AllocMem=0 FreeMem=1336595 Sockets=2 Boards=1
   State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=il
   CfgTRES=cpu=80,mem=1480000M,billing=80,gres/gpu=10,gres/gpu:2080ti=10
   AllocTRES=

Here, you can see what resources exist on the node and how many are currently allocated.

QoS controls time limits, GPU limits, and priority inside the il partition. You can list the live QoS settings with:

roed@ilc:~$ sacctmgr show qos format=Name,Priority,MaxWall,MaxTRESPU,Flags%20,Preempt,PreemptMode
      Name   Priority     MaxWall     MaxTRESPU                Flags    Preempt PreemptMode
---------- ---------- ----------- ------------- -------------------- ---------- -----------
    normal          0                                                               cluster
       dev          0                                                               cluster
        il       1000  7-00:00:00 gres/gpu:108+          DenyOnLimit      il-lo     requeue
il-intera+       1500    12:00:00    gres/gpu=2          DenyOnLimit      il-lo     requeue
     il-lo        100 21-00:00:00  gres/gpu=100                                     cluster

The table above truncates long fields. To see the full per-GPU-type limits, use parseable output:

roed@ilc:~$ sacctmgr -n -P show qos format=Name,Priority,MaxWall,MaxTRESPU,Flags,Preempt,PreemptMode
il|1000|7-00:00:00|gres/gpu:1080=10,gres/gpu:2080ti=10,gres/gpu:a100=10,gres/gpu:b200=2,gres/gpu:rtx8000=10,gres/gpu=10|DenyOnLimit|il-lo|requeue
il-interactive|1500|12:00:00|gres/gpu=2|DenyOnLimit|il-lo|requeue
il-lo|100|21-00:00:00|gres/gpu=100|||

The important operational rule is:

• Use –partition il for student jobs.
• Use –qos il-interactive for short interactive work.
• Use –qos il or omit –qos for normal jobs.
• Use –qos il-lo for low-priority jobs that can be requeued.