使用spark-shell操作Carbondata步骤讲解

0x0 准备

下载spark，解压

下载carbondata并编译

将编译好的carbondata jar包放入spark的jars文件夹中

如果要对HDFS的文件进行增删改，最好加入环境变量：

HADOOP_USER_NAME=hdfs

可以启动spark-shell了

0x1 以local模式启动spark-shell

1. 启动脚本

./bin/spark-shell

2. scala语句

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
import org.apache.spark.SparkConf
import org.apache.carbondata.core.util.CarbonProperties
CarbonProperties.getInstance().addProperty("carbon.lock.type", "HDFSLOCK")
val conf = new SparkConf().setMaster("spark://192.168.0.181:7077").set("spark.cores.max", "4")
val carbon = SparkSession.builder().config(conf).config("hive.metastore.uris","thrift://192.168.0.181:9083").getOrCreateCarbonSession("hdfs://192.168.0.181:8020/opt")

val carbon = SparkSession.builder().config("hive.metastore.uris","thrift://192.168.0.181:9083").getOrCreateCarbonSession("hdfs://192.168.0.181:8020/opt")

carbon.sql("show tables").show
carbon.sql("select * from event_log").show

carbon.sql("select * from event_log").coalesce(1).write().option("header", "true").option("sep", ",").option("escapeQuotes", "false").mode("overwrite").csv("/root/csv");

carbon.sql("update event_log_test set (content)=('test02') where eventLogId='EVENT_LOG_ID_10'")

0x2 以StandAlone模式启动spark-shell

1. 启动脚本

./bin/spark-shell --master spark://192.168.0.181:7077 --total-executor-cores 4 --jars 需要的额外的jar路径

2. scala语句

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
import org.apache.spark.SparkConf
import org.apache.carbondata.core.util.CarbonProperties
CarbonProperties.getInstance().addProperty("carbon.lock.type", "HDFSLOCK")
val carbon = SparkSession.builder().config(sc.getConf).config("hive.metastore.uris","thrift://192.168.0.181:9083").getOrCreateCarbonSession("hdfs://192.168.0.181:8020/opt")
carbon.sql("show tables").show

0x3 其他

============性能测试=============

import org.apache.spark.sql.catalyst.util._

benchmark{要执行的语句}

例如：

benchmark{carbon.sql(“select * from user”)}

可以测得大括号内执行的时间