MR汇聚工具步骤

---------------------------------mr汇聚工具步骤-----------------------------------
-- 1、需要连上141服务器 用户:root 密码:richr00t
-- 2、【mr汇聚工具】
--3、位置 141目录下进入后切换fast用户 su fast
/home/fast/luohai/projects

------------------------第一步首先需要创建外部表-----------------------------

【mr定位表建立工具】
--4、建立mr定位数据的外部表
/home/elastic-combine/yyl/mro_load_hive
--5、执行建立外部表
sh ./mro_load_hive_jingdui.sh 地市名(大写)、日期(如:20180921)

--6、进入mr定位表建立工具目录下创建所需要创建的地市外部表
如: sh ./mro_load_hive_jingdui.sh zhumadian 20180921

--7、需要删除外部表可以执行
sh ./mro_delete_jingdui.sh
如：sh ./mro_delete_jingdui.sh zhumadian 20180921

-----------------------创建完外部表需要配置文件----------------------
--9、需要配置的文件有conf、run
bin:大脑
conf:配置文件
jars:存放java包
log: 存放日志
origin:传参工具
run:存放shell脚本
sql:存放元数据表

--10、--------------------第二步配置conf目录下-----------------------------位置/home/fast/luohai/projects/conf
所需要配置的文件有tot.conf、tot_road.conf、tot_scene.conf进入修改文件需要vim
栅格 道路 场景

如 vim tot.conf vim是一个编译器只能编译文件也可以创建文件

1.tot.conf (cityname、date)需要修改的文件
###可以修改也可以不用修改(由素绢发来的邮件为准)
cityname=zhengzhou
###日期必须修改(由素绢发来的邮件为准)
date=20180921,20180925

2.tot_road.conf(cityname、date)需要修改的文件
###可以修改也可以不用修改(由素绢发来的邮件为准)
cityname=zhengzhou
###日期必须修改(由素绢发来的邮件为准)
date=20180921,20180925

3.tot_scene.conf
###可以修改也可以不用修改(由素绢发来的邮件为准)
cityname=zhengzhou
###日期必须修改(由素绢发来的邮件为准)
date=20180921,20180925

---------------------第三部配置run目录下------------------------------------位置/home/fast/luohai/projects/run
配置文件run_by_city.sh下的

###必须要修改的地市名(由素绢发来的邮件为准)
也可以多个地市跑 以空格来分隔
citynames="地市名 地市名"

sh $app_home/run/runmain.sh tot $cityname (栅格级)
sh $app_home/run/runmain.sh tot_scene $cityname (场景级)
sh $app_home/run/runmain.sh tot_road $cityname (道路级)

须知:每跑一个地市需要全部打开这三个栅格级、场景级、道路级的文件
配置好开启run_by_city.sh
sh ./run_by_city.sh 开启后会出现以下现象

info] 2018-10-19 10:11:28 using project:tot
[info] 2018-10-19 10:11:28 cities to run:luoyang
[info] 2018-10-19 10:11:28 running:luoyang
[info] 2018-10-19 10:11:28 fetch confgure cityname:luoyang,date:20180921,20180925,database:fastdo_lte_wk5
[info] 2018-10-19 10:11:28 log file is /home/fast/luohai/projects/log/tot.log
[info] 2018-10-19 10:11:28 render finish.
[info] 2018-10-19 10:11:28 start run spark:tot

###开启后出现start run spark:tot就是正在跑数据
一般跑完需要好几个小时我们要时刻监控着

###跑完一般会出现两种情况成功和报错
###第一种运行成功如下:succeed
using project:tot
[info] 2018-10-15 18:07:07 cities to run:zhoukou
[info] 2018-10-15 18:07:07 running:zhoukou
[info] 2018-10-15 18:07:07 fetch confgure cityname:zhoukou,date:20180921,20180925,database:fastdo_lte_wk5
[info] 2018-10-15 18:07:07 log file is /home/fast/luohai/projects/log/tot.log
[info] 2018-10-15 18:07:07 render finish.
[info] 2018-10-15 18:07:07 start run spark:tot
[info] 2018-10-15 21:30:40 succeed end run spark:tot #出现succeed就是没有问题
[info] 2018-10-15 21:30:40 using project:tot_scene
[info] 2018-10-15 21:30:40 cities to run:zhoukou
[info] 2018-10-15 21:30:40 running:zhoukou
[info] 2018-10-15 21:30:40 fetch confgure cityname:zhoukou,date:20180921,20180925,database:fastdo_lte_wk5
[info] 2018-10-15 21:30:40 log file is /home/fast/luohai/projects/log/tot_scene.log
[info] 2018-10-15 21:30:40 render finish.
[info] 2018-10-15 21:30:40 start run spark:tot_scene
[info] 2018-10-15 23:03:08 succeed end run spark:tot_scene #出现succeed就是没有问题
[info] 2018-10-15 23:03:08 using project:tot_road
[info] 2018-10-15 23:03:08 cities to run:zhoukou
[info] 2018-10-15 23:03:08 running:zhoukou
[info] 2018-10-15 23:03:08 fetch confgure cityname:zhoukou,date:20180921,20180925,database:fastdo_lte_wk5
[info] 2018-10-15 23:03:08 log file is /home/fast/luohai/projects/log/tot_road.log
[info] 2018-10-15 23:03:08 render finish.
[info] 2018-10-15 23:03:08 start run spark:tot_road
[info] 2018-10-15 23:38:47 succeed end run spark:tot_road #出现succeed就是没有问题

###第二种情况 failed

[info] 2018-10-15 18:07:07 cities to run:zhoukou
[info] 2018-10-15 18:07:07 running:zhoukou
[info] 2018-10-15 18:07:07 fetch confgure cityname:zhoukou,date:20180921,20180925,database:fastdo_lte_wk5
[info] 2018-10-15 18:07:07 log file is /home/fast/luohai/projects/log/tot.log
[info] 2018-10-15 18:07:07 render finish.
[info] 2018-10-15 18:07:07 start run spark:tot
[info] 2018-10-15 21:30:40 failed end run spark:tot #出现failed就是有问题
[info] 2018-10-15 21:30:40 using project:tot_scene
[info] 2018-10-15 21:30:40 cities to run:zhoukou
[info] 2018-10-15 21:30:40 running:zhoukou
[info] 2018-10-15 21:30:40 fetch confgure cityname:zhoukou,date:20180921,20180925,database:fastdo_lte_wk5
[info] 2018-10-15 21:30:40 log file is /home/fast/luohai/projects/log/tot_scene.log
[info] 2018-10-15 21:30:40 render finish.
[info] 2018-10-15 21:30:40 start run spark:tot_scene
[info] 2018-10-15 23:03:08 failed end run spark:tot_scene #出现failed就是有问题
[info] 2018-10-15 23:03:08 using project:tot_road
[info] 2018-10-15 23:03:08 cities to run:zhoukou
[info] 2018-10-15 23:03:08 running:zhoukou
[info] 2018-10-15 23:03:08 fetch confgure cityname:zhoukou,date:20180921,20180925,database:fastdo_lte_wk5
[info] 2018-10-15 23:03:08 log file is /home/fast/luohai/projects/log/tot_road.log
[info] 2018-10-15 23:03:08 render finish.
[info] 2018-10-15 23:03:08 start run spark:tot_road
[info] 2018-10-15 23:38:47 failed end run spark:tot_road #出现failed就是有问题

出现failed我们需要检查是不是外部表没有创建好，我们需要一一排查

--------------------------------------最后一步去hive里执行龙哥写好的脚本-----------------------------------------
1.进入hive执行命令
su hive 切换成hive用户
2.在执行hive就成功进入hive了
hive
3.进入wk5库
show databases;查看拥有的数据库
use fastdo_lte_wk5;进入wk5库

4.需要修改龙哥写好的脚本的地市名
#在去wk5库进行汇聚