mesos+zk+Marathon实现对docker容器的管理
1、系统环境:CentOS Linux release 7.6.1810 (Core)
五台机器
[aaa@qq.com mesos]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.81 m1 #安装mesos,zk
192.168.137.82 m2 #安装mesos,zk
192.168.137.83 m3 #安装mesos,zk
192.168.137.85 m5 #安装mesos,docker
192.168.137.86 m6 #安装mesos,docker
2、配置必要环境，五台机器全部执行
关闭防火墙,selinux,ssh加速等，执行
[aaa@qq.com init]# cat 1.sh
#!/bin/bash
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
sed -i 's/GSSAPIAuthentication yes /GSSAPIAuthentication no/' /etc/ssh/sshd_config
sed -i '/#UseDNS yes/a\UseDNS no' /etc/ssh/sshd_config
service firewalld stop
chkconfig firewalld off
必要的文件存放在这个路径，方便脚本调用
[aaa@qq.com mesos]# ls /shared/app/install/tar.gz/mesos/
marathon-0.15.2.tgz mesos-1.6.1.tar.gz pip-1.5.4.tar.gz
mesos-0.25.0.tar.gz others zookeeper-3.4.12.tar.gz
配置JAVA环境，注意下载jdk-8u181-linux-x64.tar.gz文件的存放路径，执行
[aaa@qq.com mesos]# cat java.sh
#!/bin/bash
cd /shared/app
if [ ! -d /tmp/java ];then
mkdir -p /tmp/java
fi
tar -zxvf jdk-8u181-linux-x64.tar.gz -C /tmp/java/
if [ -d /usr/local/java ];then
mv /usr/local/java /usr/local/javaold
fi
mv /tmp/java/jdk1.8.0_181 /usr/local/java
cp -r /etc/profile /etc/profile.bak
echo "export JAVA_HOME=/usr/local/java" >> /etc/profile
echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> /etc/profile
echo "export CLASSPATH=\$JAVA_HOME/jre/lib/ext:\$JAVA_HOME/lib/tools.jar" >> /etc/profile
source /etc/profile
java -version
安装必要的软件
[aaa@qq.com mesos]# cat yum.sh
#!/bin/bash
yum -y groupinstall "Development Tools"
wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
cat >> /etc/yum.repos.d/wandisco-svn.repo <<EOF
[WANdiscoSVN]
name=WANdisco SVN Repo 1.9
enabled=1
baseurl=http://opensource.wandisco.com/centos/7/svn-1.9/RPMS/$basearch/
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF
sleep 3
yum install -y apache-maven python-devel python-six python-virtualenv java-1.8.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel apr-devel subversion-devel apr-util-devel cyrus-sasl-md5
3、安装mesos，我这次是编译安装，事实证明非常慢，影响效率
[aaa@qq.com mesos]# cat mesos.sh
#!/bin/bash
cd /shared/app/install/tar.gz/mesos/
tar -zxvf mesos-0.25.0.tar.gz -C /tmp/
cd /tmp/mesos-0.25.0
mkdir build
cd build
../configure
make
make check
make install
echo "export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so" >> /etc/profile
echo "export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so" >> /etc/profile
source /etc/profile
4、安装zookeeper
[aaa@qq.com mesos]# cat zk.sh
#!/bin/bash
if [ ! -d /zk ];then
mkdir -p /zk
fi
cd /shared/app/install/tar.gz/mesos/
tar -zxvf zookeeper-3.4.12.tar.gz -C /zk
mv /zk/zookeeper-3.4.12 /zk/zookeeper
cp /zk/zookeeper/conf/zoo_sample.cfg /zk/zookeeper/conf/zoo.cfg
mkdir /zk/zookeeper/data
mkdir /zk/zookeeper/datalog
sed -i 's/dataDir\=\/tmp\/zookeeper/dataDir\=\/zk\/zookeeper\/data/' /zk/zookeeper/conf/zoo.cfg
echo "dataLogDir=/zk/zookeeper/datalog" >> /zk/zookeeper/conf/zoo.cfg
echo "server.1=192.168.137.81:2888:3888" >> /zk/zookeeper/conf/zoo.cfg
echo "server.2=192.168.137.82:2888:3888" >> /zk/zookeeper/conf/zoo.cfg
echo "server.3=192.168.137.83:2888:3888" >> /zk/zookeeper/conf/zoo.cfg
echo $1 > /zk/zookeeper/data/myid
/zk/zookeeper/bin/zkServer.sh start /zk/zookeeper/conf/zoo.cfg
执行的时候带参数执行，m1,m2,m3三台使用1,2,3,的ID,sh -x zk.sh 1, sh -x zk.sh 2, sh -x zk.sh 3 这样执行
安装完以后查看状态，有一个为leader，两个为follower
[aaa@qq.com zookeeper]# /zk/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /zk/zookeeper/bin/../conf/zoo.cfg
Mode: leader
[aaa@qq.com ~]# /zk/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /zk/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[aaa@qq.com ~]# /zk/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /zk/zookeeper/bin/../conf/zoo.cfg
Mode: follower
5、m1,m2,m3都启动mesos-master服务，会占用一个终端
mesos-master --work_dir=/mesos/data --log_dir=/mesos/logs --no-hostname_lookup --ip=0.0.0.0 --zk=zk://192.168.137.81:2181/mesos --quorum=1
mesos-master --work_dir=/mesos/data --log_dir=/mesos/logs --no-hostname_lookup --ip=0.0.0.0 --zk=zk://192.168.137.82:2181/mesos --quorum=1
mesos-master --work_dir=/mesos/data --log_dir=/mesos/logs --no-hostname_lookup --ip=0.0.0.0 --zk=zk://192.168.137.83:2181/mesos --quorum=1
此时访问mesos会自动跳转到master的站点
到master的站点部署Marathon，本实验为m1机器192.168.137.81
Marathon版本得和mesos相兼容
cd /shared/app/install/tar.gz/mesos/
tar -zxvf marathon-1.4.6.tgz -C /tmp/
mv /tmp/marathon-1.4.6/ /marathon
cd /marathon/
./bin/start --hostname 192.168.137.81 --master zk://192.168.137.81:2181,192.168.137.82:2181,192.168.137.83:2181/mesos --http_address 0.0.0.0
访问界面如下：
6、m5,m6启动mesos-slave,两台机器都一样
先安装好docker
mkdir -p /mesos/data /mesos/logs
mesos-slave --containerizers="mesos,docker" --work_dir=/mesos/data --log_dir=/mesos/logs --master=zk://192.168.137.81:2181,192.168.137.82:2181,192.168.137.83:2181/mesos --no-hostname_lookup --ip=0.0.0.0
slave节点加进来以后mesos上面可以看得到
7、推送任务至marathon
写好docker启动的参数，保存为json格式，在任意机器上面post至marathon的api接口完成推送
[aaa@qq.com tmp]# cat ng.json
{
"id":"/nginx",
"container":{
"type":"DOCKER",
"docker":{
"image":"nginx",
"network":"HOST",
"parameters":[],
"privileged":false,
"forcePullImage":false
}
},
"cpus":0.5,
"mem":256,
"instances":1
}
[aaa@qq.com tmp]# curl -X POST http://192.168.137.81:8080/v2/apps -d @ng.json -H "Content-type: application/json"
{"id":"/nginx","cmd":null,"args":null,"user":null,"env":{},"instances":1,"cpus":0.5,"mem":256,"disk":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"ports":[0],"requirePorts":false,"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"nginx","network":"HOST","privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":1,"maximumOverCapacity":1},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2019-05-30T08:44:34.231Z","tasksStaged":0,"tasksRunning":0,"tasksHealthy":0,"tasksUnhealthy":0,"deployments":[{"id":"288543d7-3770-49bb-992f-8af88992ee6f"}],"tasks":[]}
实际操作的时候，失败了很多次，发现是slave服务器docker配置的原因
报错信息如下：
I0530 16:36:03.093997 46571 exec.cpp:134] Version: 0.25.0
I0530 16:36:03.130642 46578 exec.cpp:208] Executor registered on slave 2219633f-9257-4b0d-be8b-0f82ed10be18-S0
/usr/bin/docker-current: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/673b8393af503f0e5832ad052fee0c2b6a5cd341f0e7b8955b5cd96f525b95ee-init/merged: invalid argument.
See '/usr/bin/docker-current run --help'.
W0530 16:36:03.130642 46580 logging.cpp:81] RAW: Received signal SIGTERM from process 6373 of user 0; exiting处理过程如下：
[aaa@qq.com init.d]# service docker stop
Redirecting to /bin/systemctl stop docker.service
[aaa@qq.com init.d]# rm -rf /var/lib/docker
vi /etc/sysconfig/docker-storage
改成：
DOCKER_STORAGE_OPTIONS="--storage-driver overlay"
vi /etc/sysconfig/docker
去掉option的–selinux-enabled
[aaa@qq.com init.d]# service docker start
Redirecting to /bin/systemctl start docker.service
再次推送，其实通过WEB界面推送也是一样的
从安全的角度考虑，MESOS应做好用户认证和授权，实验中为测试效果，没有使用认证
8、验证任务的执行
在MESOS上面查看到任务在192.168.137.86机器执行
192.168.137.86机器上面容器已经启动
[aaa@qq.com tmp]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39d83df036de nginx "nginx -g 'daemon ..." 2 hours ago Up 2 hours mesos-2219633f-9257-4b0d-be8b-0f82ed10be18-S0.83e4ace4-25ca-47ad-a98c-510851e8e8ed
验证访问
推荐阅读
-
mesos+zk+Marathon实现对docker容器的管理
-
如何使用Spring+redis实现对session的分布式管理
-
spring源码学习系列4.1-spring实现对ibatis的事务管理 博客分类： spring spring
-
docker容器中布置静态网站的实现
-
Android中对已安装应用的管理实现
-
Docker容器中挂载NFS共享目录的实现
-
客户关系管理系统中实现对客户信息标记颜色的处理
-
基于spring-boot和docker-java实现对docker容器的动态管理和监控功能[附完整源码下载]
-
docker overlay实现跨主机的容器互通的方法
-
Docker容器监控及日志管理实现过程解析