Hadoop Cluster
- Total CPU cores: 320
- Total memory: 2560 GB
- Raw storage: 800 TB
Hadoop servers
| Server | OS | Memory [GB] | CPU Type | Arch | CPU Speed | CPUs | Cores | Threads | Local Storage [TB] | Login | Description | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ilhadoop1, ilhadoop2 | Ubuntu 16.04 | 64 | Opteron 6276 | 64 bit | 2.3 | 2 | 32 | 32 | 2 | CS | !JobTracker, !NameNode | |
| ilh01-ilh40 | Ubuntu 16.04 | 64 | Opteron 6320 | 64 bit | 2.8 | 1 | 8 | 8 | 8 | N/A | !TaskTracker, !DataNode | |
Using Hadoop cluster
Job submission from servers:
- madmax1.stanford.edu
- madmax2.stanford.edu
- madmax3.stanford.edu
- madmax4.stanford.edu
user@madmax:~$ hadoop fs -ls / Found 5 items drwxr-xr-x - hdfs hadoop 0 2018-09-12 13:33 /dataset drwxr-xr-x - hdfs hadoop 0 2018-01-11 09:07 /lost+found drwxrwxrwx - yarn hadoop 0 2018-06-30 10:37 /tmp drwxr-xr-x - hdfs hadoop 0 2018-09-12 13:45 /user drwxr-xr-x - yarn hadoop 0 2014-09-18 16:10 /var
Submitting an example job with spark
user@madmax:~$ spark-submit --deploy-mode client \
--class org.apache.spark.examples.JavaSparkPi \
/opt/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar 10