在IPython Notebook使用Spark

在IPython Notebook使用Spark

PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS="notebook" pyspark

IPython Notebook 运行在hadoop Yarn-client模式

start-all.sh

PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS="notebook" HAHOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop MASTER=yarn-client pyspark

使用IPython Notebook在Spark Stand Alone模式运行

start-all.sh

/usr/local/spark/sbin/start-all.sh 

PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS="notebook" MASTER=spark://master:7077 pyspark --num-executors 1 --total-executor-cores 2 --executor-memory 512m