kafka集群的搭建

1、准备环境

• 使用VMware安装3台Linux服务器

  • mini版本
  • hostname 依次为 node01、node02、node03
  • 关闭防火墙
    • 每台机器都操作一遍
    • service iptables stop && chkconfig iptables off
  • 配置网卡

  • 配置/etc/hosts文件

    192.168.140.128 node01 zk01 kafka01
    192.168.140.129 node02 zk02 kafka02
    192.168.140.130 node03 zk03 kafka03
    scp /etc/hosts node02:/etc/
    scp /etc/hosts node03:/etc/
    

  • 配置Yum

    yum install -y lrzsz
    yum install -y wget
    mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
    cd /etc/yum.repos.d
    wget http://mirrors.163.com/.help/CentOS6-Base-163.repo
    yum clean all && yum makecache
    

  • 配置免密登录

    yum -y install openssh-clients
    ssh-****** 四个回车
    ssh-copy-id node1
    ssh-copy-id node2
    ssh-copy-id node3
    

2、安装JDK&ZK&KAFKA

• 安装JDK

  mkdir -p /export/servers
  mkdir -p /export/software
  mkdir -p /export/logs
  mkdir -p /export/data
  mkdir -p /export/data/zk
  mkdir -p /export/data/kafka
  mkdir -p /export/logs/zk
  -
  cd /export/software/
  rz 选择 jdk-8u141-linux-x64.tar.gz
  tar -zxvf jdk-8u141-linux-x64.tar.gz -C ../servers/
  cd ../servers/
  mv jdk1.8.0_141 jdk
  source /etc/profile
  -
  export JAVA_HOME=/export/servers/jdk
  export JRE_HOME=${JAVA_HOME}/jre
  export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
  export PATH=${JAVA_HOME}/bin:$PATH
  -
  scp -r jdk node02:/export/servers/
  scp -r jdk node03:/export/servers/
  scp /etc/profile node02:/etc/
  scp /etc/profile node03:/etc/
  -
  source /etc/profile on node01、node02、node03
  java -version on node01 on node01、node02、node03
  

• 安装zookeeper

  cd /export/software/
  wget http://219.238.7.73/files/703900000A354B91/apache.fayea.com/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz
  tar -zxvf zookeeper-3.4.9.tar.gz -C ../servers/
  cd ../servers/
  mv zookeeper-3.4.9/ zk
  -
  scp -r zk node02:/export/servers/
  scp -r zk node03:/export/servers/
  -
  touch /export/data/zk/myid on node01、node02、node03
  echo 1 > /export/data/zk/myid on node01
  echo 2 > /export/data/zk/myid on node02
  echo 3 > /export/data/zk/myid on node03
  -
  cd /export/servers/zk/conf/
  touch zoo.cfg 
  vi zoo.cfg
  -
  tickTime=2000
  initLimit=10
  syncLimit=5
  dataDir=/export/data/zk
  dataLogDir=/export/logs/zk
  clientPort=2181
  server.1=node01:2887:3887
  server.2=node02:2887:3887
  server.3=node03:2887:3887
  -
  scp zoo.cfg node02:$PWD
  scp zoo.cfg node03:$PWD
  -
  -
  zkServer.sh start on node01、node02、node03
  zkServer.sh status on node01、node02、node03
  

• 安装Kafka

  cd /export/software/
  wget http://219.238.7.67/files/518200000AE89181/mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz
  tar -zxvf kafka_2.11-1.0.0.tgz -C ../servers/
  cd ../servers/
  mv kafka_2.11-1.0.0 kafka
  -
  scp -r kafka node03:/export/servers/
  scp -r kafka node03:/export/servers/
  -
  cd /export/servers/kafka/config/
  -
  修改配置文件
  cd /export/servers/kafka/config

• vi server.properties
  

  修改三个地方:

  • broker.id

    • 每个kafka实例都具备一个唯一的id

  • log.dirs

    • Kafka用来存放消息的路径，需要在每台机器上创建

  • zookeeper.connect

    • Kafka信息存放在zookeeper中，需要制定zookeeper地址。

    • 每个Kafka实例启动的时候，都会将自己的信息注册到zookeeper中。

  最后启动kafka

• jps查看进程