Storm实战 (1) storm1.0.0集群安装

程序员文章站 2022-06-16 15:26:08

...

storm1.0.0集群安装

1.storm集群组成

Storm实战 (1) storm1.0.0集群安装

博客分类： storm storm集群1.0.0

storm集群类似于Hadoop(1.x)集群，在Hadoop(1.x)集群中任务叫"MapReduce jobs"而在storm集群中任务叫"topologies"，"MapReduce jobs"和"topologies"的最大区别是"MapReduce jobs"执行后就结束了，但是"topologies"一直执行直到你强制关闭它。storm集群和Hadoop(1.x)集群对比如下：

	Hadoop(1.x)	Storm
集主节点进程	JobTracker	Nimbus
工作节点进程	TaskTracker	Supervisor
应用名称	Job	Topology
API接口	Mapper/Reducer	Spout/Bolt
使用场景	离线数据分析处理	实时数据分析处理

storm集群中的节点分为如下三类：

master nodes :主节点上运行的进程叫Nimbus，Nimbus主要负责分发客户端提交的代码到集群中，负责分配任务以及监控任务的执行。
worker nodes :工作节点上运行的进程叫Supervisor，Supervisor监听分发给自己的任务代码，启动和关闭工作进程来执行Nimbus分发的任务，每个工作进程执行有很多个topology组成的任务，一个被执行的topology由许多分布于集群机器上的工作进程组成。
zookeeper nodes： Nimbus和Supervisor节点之间所有的协调工作是通过Zookeeper集群来实现的。此外，Nimbus和Supervisor进程都是快速失败（fail-fast)和无状态（stateless）的；Storm集群所有的状态要么在Zookeeper集群中，要么存储在本地磁盘上。这意味着你可以用kill -9来杀死Nimbus和Supervisor进程，它们在重启后可以继续工作。这个设计使得Storm集群拥有不可思议的稳定性。

2.storm集群搭建

搭建Zookeeper集群
安装Storm依赖库
下载并解压Storm发布版本
修改storm.yaml配置文件
启动Storm各个后台进程

2.1 搭建Zookeeper集群

1）下载并解压zookeeper3.4.6

#下载zookeeper-3.4.6.tar.gz到/opt并解压解压
cd /opt
tar -zxvf zookeeper-3.4.6.tar.gz

2）在集群中各个节点中配置/etc/hosts，内容如下：

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.202.131 master
192.168.202.132 slavery01
192.168.202.133 slavery02

3）在集群中各个节点中创建zookeeper数据文件

sudo rm -r /home/hadoop/zookeeper
cd  /home/hadoop
mkdir zookeeper

4）在hostname=master机器上配置zoo.cfg，将/opt/zookeeper-3.3.4/conf目录下面的 zoo_sample.cfg修改为zoo.cfg，配置文件内容如下所示:

initLimit=10
syncLimit=5
dataDir=/home/hadoopmanage/zookeeper
clientPort=2181
server.1=master:2888:3888
server.2=slavery01:2888:3888
server.3=slavery02:2888:3888

5）远程复制分发安装文件

scp -r /opt/zookeeper-3.3.4 hadoop@slavery01:/opt/
scp -r /opt/zookeeper-3.3.4 hadoop@slavery02:/opt/

6）在集群中各个节点设置myid必须为数字

#使用ssh是首先保证你集群中节点都做了免秘钥登录
ssh master
echo "1" > /home/hadoop/zookeeper/myid  
ssh slavery01
echo "2" > /home/hadoop/zookeeper/myid  
ssh slavery02
echo "3" > /home/hadoop/zookeeper/myid

7）启动ZooKeeper集群

cd /opt/zookeeper-3.4.6
bin/zkServer.sh start

8）查看单机ZooKeeper是leader还是follower

cd /opt/zookeeper-3.4.6
bin/zkServer.sh status

9）停止ZooKeeper集群

cd /opt/zookeeper-3.4.6
bin/zkServer.sh stop

10）用客户端查看zookeeper上数据

cd /opt/zookeeper-3.4.6/  
bin/zkCli.sh -server master:2181,slavery01:2181,slavery02:2181

[hadoop@master storm-1.0.0]$ cd /opt/zookeeper-3.4.6/  
[hadoop@master zookeeper-3.4.6]$ bin/zkCli.sh -server master:2181,slavery01:2181,slavery02:2181  
Connecting to master:2181,slavery01:2181,slavery02:2181
2016-05-02 16:39:29,880 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-05-02 16:39:29,889 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=master
2016-05-02 16:39:29,889 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_65
2016-05-02 16:39:29,902 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2016-05-02 16:39:29,903 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/opt/jdk1.7.0_65/jre
2016-05-02 16:39:29,903 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper-3.4.6/bin/../build/classes:/opt/zookeeper-3.4.6/bin/../build/lib/*.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/opt/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/opt/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.6/bin/../conf:.:/opt/jdk1.7.0_65/lib/dt.jar:/opt/jdk1.7.0_65/lib/tools.jar
2016-05-02 16:39:29,903 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/opt/hadoop-2.7.1/lib/native/:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-05-02 16:39:29,904 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2016-05-02 16:39:29,904 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2016-05-02 16:39:29,904 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2016-05-02 16:39:29,904 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2016-05-02 16:39:29,905 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=2.6.32-358.el6.x86_64
2016-05-02 16:39:29,905 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=hadoop
2016-05-02 16:39:29,905 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/hadoop
2016-05-02 16:39:29,906 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/zookeeper-3.4.6
2016-05-02 16:39:29,909 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=master:2181,slavery01:2181,slavery02:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@8afbefd
Welcome to ZooKeeper!
2016-05-02 16:39:30,290 [myid:] - INFO  [main-SendThread(master:2181):ClientCnxn$SendThread@975] - Opening socket connection to server master/192.168.202.131:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-02 16:39:30,350 [myid:] - INFO  [main-SendThread(master:2181):ClientCnxn$SendThread@852] - Socket connection established to master/192.168.202.131:2181, initiating session
JLine support is enabled
2016-05-02 16:39:31,469 [myid:] - INFO  [main-SendThread(master:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server master/192.168.202.131:2181, sessionid = 0x154701cef030003, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: master:2181,slavery01:2181,slavery02:2181(CONNECTED) 0]

查看zookeeper数据根目/录和数据目录/storm下数据：

[zk: master:2181,slavery01:2181,slavery02:2181(CONNECTED) 0] ls /
[storm, hbase, zookeeper]
[zk: master:2181,slavery01:2181,slavery02:2181(CONNECTED) 1] ls /storm
[backpressure, workerbeats, nimbuses, supervisors, errors, logconfigs, storms, assignments, leader-lock, blobstore]
[zk: master:2181,slavery01:2181,slavery02:2181(CONNECTED) 2]

2.2 安装Storm依赖库

1） jdk安装(官网要求是1.6或者以上版本，这里安装1.7)

1.卸载linux自带的jdk环境
    1）首先用命令java -version 查看系统中原有的java版本
    2）然后用用  rpm -qa | gcj 命令查看具体的信息
    3）最后用    rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64卸载
2.安装jdk-7u65-linux-x64.gz
    1）下载jdk-7u65-linux-x64.gz放置于/opt/java/jdk-7u65-linux-x64.gz
    2）解压，输入命令tar -zxvf jdk-7u65-linux-x64.gz
    3）编辑vi /etc/profile,在文件末尾追加如下内容
        export JAVA_HOME=/opt/java/jdk1.7.0_65  
        export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar 
         export PATH=$PATH:$JAVA_HOME/bin  
    4）使配置生效，输入命令，source /etc/profile
    5）输入命令java -version，检查JDK环境是否配置成功
        2.2.2 Python安装

2） Python安装(官网要求2.6.6以上)

#安装python2.7.10
#1)在https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz下载Python-2.7.10.tgz放置于/opt下
#2)解压tar -xzf Python-2.7.10.tgz到/opt下
cd /opt
tar -xzf Python-2.7.10.tgz
#3)编译安装Python
cd /opt/Python-2.7.10
./configue,
make
make install
#4)如果不安装或者python版本过老，在后面会报错如下：
#No module named argparse
#5)Python版本查看
python -V

2.3 下载并解压Storm发布版本

1）在hostname=master机器上下载Storm发行版本

cd /opt
wget http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.0.0/apache-storm-1.0.0.tar.gz

2）在hostname=master机器上解压到目录/opt下

cd /opt
tar -zxvf apache-storm-1.0.0.tar.gz
mv apache-storm-1.0.0 storm-1.0.0

3）在hostname=master机器上修改/opt/storm-1.0.0/conf/storm.yaml配置文件

 storm.zookeeper.servers:
     - "master"
     - "slavery01"
     - "slavery02" 
 nimbus.seeds: ["master"]
 supervisor.slots.ports:
     - 6700
     - 6701
     - 6702
     - 6703
 storm.local.dir: "/home/hadoopmanage/storm/localdir/"

注意：上面配置参数前的空格符、TAB符不要去掉，否则会报如下错误信息：

        at org.apache.storm.shade.org.yaml.snakeyaml.scanner.ScannerImpl.stalePossibleSimpleKeys(ScannerImpl.java:460)
        at org.apache.storm.shade.org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:280)
        at org.apache.storm.shade.org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:225)
        at org.apache.storm.shade.org.yaml.snakeyaml.parser.ParserImpl$ParseIndentlessSequenceEntry.produce(ParserImpl.java:532)
        at org.apache.storm.shade.org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
        at org.apache.storm.shade.org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)
        at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeSequenceNode(Composer.java:203)
        at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:157)
        at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:237)
        at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159)
        at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122)
        at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105)
        at org.apache.storm.shade.org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)
        at org.apache.storm.shade.org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)
        at org.apache.storm.shade.org.yaml.snakeyaml.Yaml.load(Yaml.java:424)
        at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:290)
        at org.apache.storm.utils.Utils.readStormConfig(Utils.java:391)
        at org.apache.storm.utils.Utils.<clinit>(Utils.java:119)
        ... 39 more

4）在hostname=master机器上分发安装文件到其他节点

cd /opt
scp -r storm-1.0.0 hadoop@slavery01:/opt
scp -r storm-1.0.0 hadoop@slavery02:/opt

5) 在各个节点上增加storm集群本地存储文件，该目录被Nimbus和Supervisor进程用于存储少量状态，如jars、confs等的本地磁盘目录，需要提前创建该目录并给以足够的访问权限。然后在storm.yaml中配置该目录

mkdir -p /home/hadoopmanage/storm/localdir/

2.4 启动Storm各个后台进程

1）在hostname=master节点启动Nimbus进程服务，放置于后台运行

cd /opt/storm-1.0.0/
bin/storm nimbus >/dev/null 2>&1 &

2）在各个hostname-slavery0*节点启动Supervisor

cd /opt/storm-1.0.0/
bin/storm supervisor >/dev/null 2>&1 &

3）在hostname=master节点启动UI进程服务，放置于后台运行

cd /opt/storm-1.0.0/
bin/storm ui >/dev/null 2>&1 &

启动后打开浏览器，访问http://master:8080/index.html 或者http://192.168.202.131:8080/index.html，打开的storm ui界面如下：

Storm实战 (1) storm1.0.0集群安装

博客分类： storm storm集群1.0.0

3.向storm集群中提交任务

3.1 启动Storm Topology

storm jar mycode.jar com.test.MyTopology arg1 arg2 arg3

其中mycode.jar是包含Topology实现代码的jar包，com.test.MyTopology的main方法是Topology的入口，arg1、arg2和arg3为org.me.MyTopology执行时需要传入的参数。

3.2 停止Storm Topology

storm kill {toponame}

其中，{toponame}为Topology提交到Storm集群时指定的Topology任务名称。

查看图片附件

Storm实战 (1) storm1.0.0集群安装

storm1.0.0集群安装

1.storm集群组成

2.storm集群搭建

2.1 搭建Zookeeper集群

2.2 安装Storm依赖库

2.3 下载并解压Storm发布版本

2.4 启动Storm各个后台进程

3.向storm集群中提交任务

3.1 启动Storm Topology

3.2 停止Storm Topology

kafka与storm集群环境的安装步骤详解

kafka与storm集群环境的安装步骤详解

ElasticSearch实战系列一: ElasticSearch集群+Kinaba安装教程

Storm集群安装与部署

《微信小程序项目开发实战：用WePY、mpvue、Taro打造高效的小程序》（笔记1）WePY开发环境的安装

Koa与Node.js开发实战（1）——Koa安装搭建（视频演示）

Storm实战 (1) storm1.0.0集群安装

hive实战(1)hive安装准备

Storm实战 (1) storm1.0.0集群安装

hive实战(1)hive安装准备