欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

spark tachyon 搭建 配置

程序员文章站 2022-07-14 13:55:39
...

spark1.5.1 支持 tachyon0.7.1

 

jdk需要1.7

 

1.spark

下载spark source

http://spark.apache.org/downloads.html

 

编译spark

 

 

export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m"

mvn -Dhadoop.version=2.3.0 -DskipTests clean package

 

spark-env.sh

因为需要访问hdfs,hive,所以需要压缩lzo,和mysql

export SPARK_CLASSPATH=$SPARK_CLASSPATH:/data/hadoop/hadoop-2.3.0-cdh5.1.0/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar
export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/data/hadoop/hadoop-2.3.0-cdh5.1.0/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/data/hadoop/spark/spark-1.5.1-bin-hadoop2.3/lib/mysql-connector-java-5.1.20-bin.jar

export SPARK_PID_DIR=/data/hadoop/spark/spark-1.5.1-bin-hadoop2.3

 

写slave文件

host136
host137
host138

 

添加配置文件到conf里:

core-site.xml

hdfs-site.xml

hive-site.xml

 

修改hive-site.xml

<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>tachyon://host136:19998/warehouse</value>
</property>

 

2.tachyon

下载tachyon,http://www.tachyon-project.org/

 

编译tachyon

mvn clean package -Djava.version=1.7 -Dhadoop.version=2.3.0 -DskipTests

 

修改配置:tachyon-env.sh

  export TACHYON_RAM_FOLDER=/data/hadoop/spark/tachyon-0.7.1-2.3/data
//内存数据mount目录
fi

if [ -z "$JAVA_HOME" ]; then
  export JAVA_HOME="$(dirname $(which java))/.."
fi

export JAVA="${JAVA_HOME}/bin/java"
export TACHYON_MASTER_ADDRESS=host136

#hdfs
export TACHYON_UNDERFS_ADDRESS=hdfs://mycluster
#-Dtachyon.master.journal.folder=hdfs://mycluster/tachyon/journal/
#export TACHYON_UNDERFS_ADDRESS=${TACHYON_UNDERFS_ADDRESS:-hdfs://localhost:9000}
#使用内存大小
export TACHYON_WORKER_MEMORY_SIZE=20GB
export TACHYON_UNDERFS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem
export TACHYON_WORKER_MAX_WORKER_THREADS=2048
export TACHYON_MASTER_MAX_WORKER_THREADS=2048

export TACHYON_SSH_FOREGROUND="yes"
export TACHYON_WORKER_SLEEP="0.02"

 可以配置多level数据

export TACHYON_JAVA_OPTS+="
  -Dlog4j.configuration=file:$CONF_DIR/log4j.properties
  -Dtachyon.debug=false
  -Dtachyon.worker.tieredstore.level.max=1
  -Dtachyon.worker.tieredstore.level0.alias=MEM
  -Dtachyon.worker.tieredstore.level0.dirs.path=$TACHYON_RAM_FOLDER
  -Dtachyon.worker.tieredstore.level0.dirs.quota=$TACHYON_WORKER_MEMORY_SIZE
  -Dtachyon.underfs.address=$TACHYON_UNDERFS_ADDRESS
  -Dtachyon.underfs.hdfs.impl=$TACHYON_UNDERFS_HDFS_IMPL
  -Dtachyon.data.folder=$TACHYON_UNDERFS_ADDRESS/tmp/tachyon/data
  -Dtachyon.worker.max.worker.threads=$TACHYON_WORKER_MAX_WORKER_THREADS
  -Dtachyon.workers.folder=$TACHYON_UNDERFS_ADDRESS/tmp/tachyon/workers
  -Dtachyon.worker.memory.size=$TACHYON_WORKER_MEMORY_SIZE
  -Dtachyon.worker.data.folder=/tachyonworker/
  -Dtachyon.master.max.worker.threads=$TACHYON_MASTER_MAX_WORKER_THREADS
  -Dtachyon.master.worker.timeout.ms=60000
  -Dtachyon.master.hostname=$TACHYON_MASTER_ADDRESS
  -Dtachyon.master.journal.folder=$TACHYON_HOME/journal/
  -Dorg.apache.jasper.compiler.disablejsr199=true
  -Djava.net.preferIPv4Stack=true
"

 

写slaves,workers

host136
host137
host138

 

添加配置文件到conf:

hdfs-site.xml

core-site.xml

 

在core-site.xml中添加

<property>
  <name>fs.tachyon.impl</name>
    <value>tachyon.hadoop.TFS</value>
    </property>
    <property>
      <name>fs.tachyon-ft.impl</name>
        <value>tachyon.hadoop.TFSFT</value>
     </property>

 

 

3启动服务:

tachyon:

需要用户有sudo权限,或是用root执行tachyon的mount操作

		如果不想每次启动Tachyon都挂载一次RamFS,可以先使用命令 bin/tachyon-mount.sh Mount workers 或 bin/tachyon-mount.sh SudoMount workers 挂载好所有RamFS,然后使用 bin/tachyon-start.sh all NoMount 命令启动Tachyon。
		
		bin/tachyon format
		bin/tachyon-mount.sh SudoMount workers
		bin/tachyon-start.sh all NoMount
		bin/tachyon-stop.sh

当添加一个tachyon节点

bin/tachyon formatWorker
bin/tachyon-mount.sh  SudoMount local
bin/tachyon-start.sh  worker NoMount

 

 spark:

bin/spark-sql --master spark://host136:7077
bin/spark-shell --master spark://host136:7077

 

 4.spark操作tachyon

用spark直接通过tachyon读取hdfs数据:

var textFile = sc.textFile("tachyon://host136:19998/hdfspath");

用spark在tachyon上建hive表:

CREATE EXTERNAL TABLE `test4`(
  `aaa` string,
  `ccc` string,
  `bbb` string)
PARTITIONED BY (
  `day_id` string,
  `hour_id` string)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'tachyon://host136:19998/tmp/storm2'

 

 页面:

http://tachyonmasterIP:19999/home

http://sparkIP:8099/

 

 

 

 

 

相关标签: spark tachyon