欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识！
您现在的位置是： 首页  >  IT编程

CloudStack多管理节点设置

程序员文章站 2022-03-23 09:37:24
CloudStack多管理节点设置。使用HAProxy实现负载均衡。由于内存不足，只创建三个虚拟机用以实现CloudStack环境搭建。两个管理节点192.168.199.91manager1192.168.199.92manager2一个计算节点192.168.199.93agent负载均衡HAProxy软件设置在 agent节点。MySQL数据库也设置在agent节点。测试有二一、初始安装时设置HAProxy。二、一管理节点一计算节点部署完成运行一段时间后，加入第二个管理节点，...

CloudStack多管理节点设置。使用HAProxy实现负载均衡。
由于内存不足，只创建三个虚拟机用以实现CloudStack环境搭建。

两个管理节点
192.168.199.91 manager1
192.168.199.92 manager2

一个计算节点
192.168.199.93 agent

一个HAProxy节点
192.168.199.90 haproxy

MySQL数据库设置在agent节点

CloudStack官方建议 设置负载均衡转发的端口。

源端口 目标端口 协议类型 持久保持
80/443 8080 HTTP 是
8250 8250 TCP 是
8096 8096 HTTP 否

如果不使用HAProxy对管理节点做负载均衡。

直接使用管理节点IP。。。管理节点设置完毕
192.168.199.91 192.168.199.92 都可以访问UI界面。
全局设置 host=192.168.199.91,192.168.199.92

192.168.199.92 IP 登录UI 创建资源域

资源域无法创建，会报错

Failed to add data store: Failed to create storage pool as it is not accessible to hosts.

两个管理节点的日志文件报错如下

192.168.199.91

2020-07-16 10:47:25,886 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-27-thread-1:null) (logid:) A client/agent attempting connection from address=192.168.199.92 has presented these certificate(s):
Certificate [1] :
 Serial: 778ff50524209e6d
  Not Before:Wed Jul 15 22:36:45 CST 2020
  Not After:Sat Jul 09 10:36:45 CST 2050
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=manager2
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:[[2, manager2]]
Certificate [2] :
 Serial: 73a8f04f8a7ad2ef
  Not Before:Wed Jul 15 22:35:13 CST 2020
  Not After:Sat Jul 09 10:35:13 CST 2050
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=ca.cloudstack.apache.org
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:null
2020-07-16 10:47:25,889 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-27-thread-1:null) (logid:) Certificate ownership verification failed for client: 192.168.199.92
2020-07-16 10:47:25,892 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/192.168.199.91:8250, remote address=/192.168.199.92:34646.

192.168.199.92

2020-07-16 10:48:10,640 ERROR [c.c.u.n.Link] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/192.168.199.92:34656, remote address=/192.168.199.91:8250. The client may have invalid ca-certificates.
2020-07-16 10:48:10,641 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) Unable to connect to peer management server: 3232286555, ip: 192.168.199.91 due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '3232286555' on 192.168.199.91:8250 
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '3232286555' on 192.168.199.91:8250 
	at com.cloud.agent.manager.ClusteredAgentManagerImpl.connectToPeer(ClusteredAgentManagerImpl.java:519)
	......
	at java.lang.Thread.run(Thread.java:748)
2020-07-16 10:48:10,642 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) Seq 1-3124935191441702917: Unable to forward null
2020-07-16 10:48:10,642 WARN  [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) Resource [Host:1] is unreachable: Host 1: Unable to reach the peer that the agent is connected
2020-07-16 10:48:10,642 WARN  [c.c.r.ResourceManagerImpl] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) Unable to obtain host 1 statistics. 
2020-07-16

192.168.199.91 IP 登录UI 创建资源域

可以成功创建资源域，但是创建完成后只有192.168.199.91 访问UI时可以看到正常二级存储容量和查看系统VM 控制台。

且两个管理节点的日志文件依旧在不停报错

192.168.199.91

2020-07-16 11:40:32,613 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-37-thread-1:null) (logid:) A client/agent attempting connection from address=192.168.199.92 has presented these certificate(s):
Certificate [1] :
 Serial: 440e1b4702d33119
  Not Before:Wed Jul 15 23:31:03 CST 2020
  Not After:Sat Jul 09 11:31:03 CST 2050
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=manager2
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:[[2, manager2]]
Certificate [2] :
 Serial: 21ba59eaf4005763
  Not Before:Wed Jul 15 23:29:11 CST 2020
  Not After:Sat Jul 09 11:29:11 CST 2050
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=ca.cloudstack.apache.org
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:null
2020-07-16 11:40:32,615 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-37-thread-1:null) (logid:) Certificate ownership verification failed for client: 192.168.199.92
2020-07-16 11:40:32,618 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/192.168.199.91:8250, remote address=/192.168.199.92:45596.

192.168.199.92

2020-07-16 11:41:27,478 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:ctx-a07d9aa9) (logid:2603e149) Seq 1-2784350469621809162: Unable to forward null
2020-07-16 11:41:27,536 ERROR [c.c.u.n.Link] (StatsCollector-4:ctx-a07d9aa9) (logid:2603e149) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/192.168.199.92:45634, remote address=/192.168.199.91:8250. The client may have invalid ca-certificates.
2020-07-16 11:41:27,537 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:ctx-a07d9aa9) (logid:2603e149) Unable to connect to peer management server: 3232286555, ip: 192.168.199.91 due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '3232286555' on 192.168.199.91:8250 
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '3232286555' on 192.168.199.91:8250 
	at com.cloud.agent.manager.ClusteredAgentManagerImpl.connectToPeer(ClusteredAgentManagerImpl.java:519)
	......
	at java.lang.Thread.run(Thread.java:748)

因此 可能需要一个 负载均衡 IP 当作 host 的值。但好像也没成功

安装CloudStack时设置HAProxy。

manager1 设置

cloudstack-setup-databases cloud:123456@agent --deploy-as=root:123456
cloudstack-setup-management

manager2 设置

cloudstack-setup-databases cloud:123456@agent
cloudstack-setup-management

注，多个管理节点时数据库的配置
max_connections 等于 350 * 管理节点数量

max_connections=700

agent节点 cloudstack-agent设置同单个管理节点一致

HAProxy 设置

安装

yum -y install haproxy

设置日志文件使得 haproxy有日志记录。

编辑haproxy配置文件

vi /etc/haproxy/haproxy.cfg

global
    log         127.0.0.1 local2

编辑 rsyslog 配置文件 取消下列行注释

vi /etc/rsyslog.conf 

$ModLoad imudp
$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

并在末尾加上

local2.*                       /var/log/haproxy.log

编辑 rsyslog 文件

 vi /etc/sysconfig/rsyslog

SYSLOGD_OPTIONS="-r"

重启日志文件和haproxy

systemctl restart rsyslog
systemctl restart haproxy

HAProxy负载均衡设置

在defaults设置中加上 cookie 设置达到持久保持状态。

defaults
	......
    cookie cloud_cook  insert  nocache

cloud_cook 表示haproxy将在浏览器之上设置的cookie名，insert nocache表示cookie的设置方式。

frontend cloud_source_80
    bind *:80
    bind *:443
    use_backend cloud_dec_8080

backend cloud_dec_8080
    balance     roundrobin
    server manager1_8080 192.168.199.91:8080 cookie manager1 check inter 2000 rise 2 fall 5
    server manager2_8080 192.168.199.92:8080 cookie manager2 check inter 2000 rise 2 fall 5


frontend cloud_source_8250
    bind *:8250
	mode tcp
    use_backend cloud_dec_8250

backend cloud_dec_8250
    balance     roundrobin
	mode tcp
    server manager1_8250 192.168.199.91:8250 cookie manager1 check inter 2000 rise 2 fall 5
    server manager2_8250 192.168.199.92:8250 cookie manager2 check inter 2000 rise 2 fall 5


frontend cloud_source_8096
    bind *:8096
    use_backend cloud_dec_8096

backend cloud_dec_8096
    balance     roundrobin
    server manager1_8096 192.168.199.91:8096 check inter 2000 rise 2 fall 5
    server manager2_8096 192.168.199.92:8096 check inter 2000 rise 2 fall 5


listen stats
    bind :32700
    stats enable
    stats refresh 60s
    stats uri /
    stats hide-version
    stats auth admin:123456

cookie manager1表示haproxy 转发到192.168.199.91 时，cloud_cook的值为manager1。带有此cookie，发出的请求都会转发到192.168.199.91。这就保证了会话保持。
应该是这样。。。

重新启动改 haproxy

systemctl restart haproxy

访问 192.168.199.90 即可进入CloudStack的UI界面
修改 全局设置 host 值为 192.168.199.90

但是，不知什么原因，基本上添加主机一步会失败，不是因计算节点配置 Libvirtd失败，就是SSL问题失败。

2020-07-15 10:49:37,498 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-23-thread-1:null) (logid:) Certificate ownership verification failed for client: 192.168.199.90
2020-07-15 10:49:37,500 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/192.168.199.91:8250, remote address=/192.168.199.90:50178.

2020-07-15 10:49:52,770 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-24-thread-1:null) (logid:) Certificate ownership verification failed for client: 192.168.199.90
2020-07-15 10:49:52,771 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/192.168.199.92:8250, remote address=/192.168.199.90:45998.

似乎出现Libvirtd失败是因为网络问题。VMwareWorkstations虚拟机需要重置虚拟网络编辑器才可重新开始测试，不然就会出现Libvirtd启动失败

SSL 就真的解决不了了。。。

计算节点日志文件

2020-07-15 17:44:57,972 INFO  [cloud.agent.Agent] (main:null) (logid:) Connecting to host:192.168.199.90
2020-07-15 17:44:57,972 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.199.90:8250
2020-07-15 17:44:57,974 INFO  [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2020-07-15 17:44:58,061 ERROR [utils.nio.Link] (main:null) (logid:) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/192.168.199.93:38458, remote address=/192.168.199.90:8250. The client may have invalid ca-certificates.
2020-07-15 17:44:58,061 ERROR [utils.nio.NioClient] (main:null) (logid:) SSL Handshake failed while connecting to host: 192.168.199.90 port: 8250
2020-07-15 17:44:58,061 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.199.90 port: 8250
	at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
	at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
	at com.cloud.agent.Agent.start(Agent.java:293)
	at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
	at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
	at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
	at com.cloud.agent.AgentShell.start(AgentShell.java:512)
	at com.cloud.agent.AgentShell.main(AgentShell.java:547)
2020-07-15 17:44:58,062 INFO  [utils.exception.CSExceptionErrorCode] (main:null) (logid:) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2020-07-15 17:44:58,062 WARN  [cloud.agent.Agent] (main:null) (logid:) NIO Connection Exception  com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.199.90 port: 8250
2020-07-15 17:44:58,062 INFO  [cloud.agent.Agent] (main:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again...

但是又不想辛辛苦苦设置大半天，一点用都没有，因此，先记录到此。等以后有机会明白为什么时再来修改

本文地址：https://blog.csdn.net/dandanfengyun/article/details/107335916

相关标签： 负载均衡 云计算 云计算

上一篇： iptables（防火墙）和selinux详情

下一篇： iPhone12来电静音无法提示来电怎么办?

推荐阅读