Spark-sql结合hive使用详情
程序员文章站
2022-05-28 20:30:16
1.安装hive(配置好)
2.将配置好的hive-site.xml放入$SPARK-HOME/conf目录下
3.将mysql-xxx-connector-xx.jar放到spark集群...
1.安装hive(配置好)
2.将配置好的hive-site.xml放入$SPARK-HOME/conf目录下
3.将mysql-xxx-connector-xx.jar放到spark集群中节点的lib目录下
4.启动spark-shell时指定mysql连接驱动位置
bin/spark-shell \ --master spark://node1.itcast.cn:7077 \ --executor-memory 1g \ --total-executor-cores 2 \ --driver-class-path /usr/local/apache-hive-0.13.1-bin/lib/mysql-connector-java-5.1.35-bin.jar
4.使用sqlContext.sql调用HQL
sqlContext.sql("select * from spark.person limit 2") 或使用org.apache.spark.sql.hive.HiveContext import org.apache.spark.sql.hive.HiveContext val hiveContext = new HiveContext(sc) hiveContext.sql("select * from spark.person").show() 或者.write.mode("append").jdbc()