解压parquet格式文件到text 博客分类: parquet
程序员文章站
2024-03-26 10:49:23
...
方法一:spark python实现
import sys
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
inputpath=sys.argv[1]
outputpath=sys.argv[2]
sc = SparkContext(appName="Transform Pq to Csv")
sqlContext = SQLContext(sc)
df = sqlContext.read.parquet(inputpath)
df.select('*').save(outputpath,"com.databricks.spark.csv",delimiter='\001')
run:spark-submit --packages com.databricks:spark-csv_2.10:1.2.0 --master yarn-client read_pq.py /tmp/xing/20161115/1049 /tmp/xing/20161115/text/1049
方法二:如果是parquet表里的数据,可以通过hql查询得到数据
insert overwrite (local) inpath outputpath
select * from table1_parquet;