Spark on farm - Revision history

Tallada at 09:14, 27 April 2021

2021-04-27T09:14:34Z

Tallada at 14:40, 18 February 2019

2019-02-18T14:40:04Z

Tallada at 09:48, 16 February 2019

2019-02-16T09:48:15Z

Tallada at 15:48, 14 February 2019

2019-02-14T15:48:09Z

Tallada at 10:09, 13 February 2019

2019-02-13T10:09:01Z

Tallada: Created page with "In order to launch a Spark cluster, follow these steps: 1) Deploy a Spark installation on your ~ 2) On a UI, start a spark master using this command and write down the maste..."

2019-02-13T10:01:32Z

Created page with "In order to launch a Spark cluster, follow these steps: 1) Deploy a Spark installation on your ~ 2) On a UI, start a spark master using this command and write down the maste..."

New page

In order to launch a Spark cluster, follow these steps:

1) Deploy a Spark installation on your ~

2) On a UI, start a spark master using this command and write down the master url provided

${SPARK_HOME}/sbin/start-master.sh -h $(hostname)

3) Create a submit script like this one, substituting <VARIABLES>

SPARK_MASTER=<MASTER_URL>
NUM_CORES=1
MEMORY=4g

cd ${TMPDIR}

PYSPARK_PYTHON=<PATH_TO_VIRTUALENV_BIN_PYTHON> \
<SPARK_HOME>/bin/spark-class org.apache.spark.deploy.worker.Worker ${SPARK_MASTER} -c ${NUM_CORES} -m ${MEMORY} -d ${TMPDIR}

4) Submit the previous script to the farm

qsub -q <QUEUE> -t 1..<NUM_WORKERS> <SUBMIT_SCRIPT>

5) After a few minutes, workers should start registering to the master.

6) Create your SparkContext connecting it to the <MASTER_URL>

spark = SparkSession.builder.master(<MASTER_URL>).getOrCreate()
sc = spark.sparkContext

7) Enjoy!

@@ Line 17: / Line 17: @@
      echo "spark.executor.extraJavaOptions -Djava.io.tmpdir=${TMPDIR}" > spark-defaults.conf
      echo "spark.local.dir ${TMPDIR}" >> spark-defaults.conf
-     echo "spark.pyspark.python /nfs/pic.es/user/t/tallada/env/correlation/bin/python" >> spark-defaults.conf
+     echo "spark.pyspark.python <PYTHON_VENV_EXECUTABLE_PATH>" >> spark-defaults.conf
      <SPARK_HOME>/bin/spark-class \
      org.apache.spark.deploy.worker.Worker ${SPARK_MASTER} \

@@ Line 18: / Line 18: @@
      echo "spark.local.dir ${TMPDIR}" >> spark-defaults.conf
      echo "spark.pyspark.python /nfs/pic.es/user/t/tallada/env/correlation/bin/python" >> spark-defaults.conf
      <SPARK_HOME>/bin/spark-class \

@@ Line 16: / Line 16: @@
      echo "spark.executor.extraJavaOptions -Djava.io.tmpdir=${TMPDIR}" > spark-defaults.conf
      <SPARK_HOME>/bin/spark-class \
      org.apache.spark.deploy.worker.Worker ${SPARK_MASTER} \

@@ Line 14: / Line 14: @@
      cd ${TMPDIR}
-−
      PYSPARK_PYTHON=<PATH_TO_VIRTUALENV_BIN_PYTHON> \
-     <SPARK_HOME>/bin/spark-class org.apache.spark.deploy.worker.Worker ${SPARK_MASTER} -c ${NUM_CORES} -m ${MEMORY} -d ${TMPDIR}
+     <SPARK_HOME>/bin/spark-class \
 ) Submit the previous script to the farm

@@ Line 20: / Line 20: @@
 ) Submit the previous script to the farm
-     qsub -q <QUEUE> -t 1..<NUM_WORKERS> <SUBMIT_SCRIPT>
+     qsub -q <QUEUE> -t 1-<NUM_WORKERS> <SUBMIT_SCRIPT>
 ) After a few minutes, workers should start registering to the master.