CloudStack多管理节点设置

CloudStack多管理节点设置。使用HAProxy实现负载均衡。
由于内存不足，只创建三个虚拟机用以实现CloudStack环境搭建。

两个管理节点
192.168.199.91 manager1
192.168.199.92 manager2

一个计算节点
192.168.199.93 agent

一个HAProxy节点
192.168.199.90 haproxy

MySQL数据库设置在agent节点

CloudStack官方建议 设置负载均衡转发的端口。

源端口	目标端口	协议类型	持久保持
80/443	8080	HTTP	是
8250	8250	TCP	是
8096	8096	HTTP	否

如果不使用HAProxy对管理节点做负载均衡。

直接使用管理节点IP。。。管理节点设置完毕
192.168.199.91 192.168.199.92 都可以访问UI界面。
全局设置 host=192.168.199.91,192.168.199.92

192.168.199.92 IP 登录UI 创建资源域

资源域无法创建，会报错

Failed to add data store: Failed to create storage pool as it is not accessible to hosts.

两个管理节点的日志文件报错如下

192.168.199.91

2020-07-16 10:47:25,886 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-27-thread-1:null) (logid:) A client/agent attempting connection from address=192.168.199.92 has presented these certificate(s):
Certificate [1] :
 Serial: 778ff50524209e6d
  Not Before:Wed Jul 15 22:36:45 CST 2020
  Not After:Sat Jul 09 10:36:45 CST 2050
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=manager2
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:[[2, manager2]]
Certificate [2] :
 Serial: 73a8f04f8a7ad2ef
  Not Before:Wed Jul 15 22:35:13 CST 2020
  Not After:Sat Jul 09 10:35:13 CST 2050
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=ca.cloudstack.apache.org
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:null
2020-07-16 10:47:25,889 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-27-thread-1:null) (logid:) Certificate ownership verification failed for client: 192.168.199.92
2020-07-16 10:47:25,892 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/192.168.199.91:8250, remote address=/192.168.199.92:34646.

192.168.199.92

2020-07-16 10:48:10,640 ERROR [c.c.u.n.Link] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/192.168.199.92:34656, remote address=/192.168.199.91:8250. The client may have invalid ca-certificates.
2020-07-16 10:48:10,641 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) Unable to connect to peer management server: 3232286555, ip: 192.168.199.91 due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '3232286555' on 192.168.199.91:8250 
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '3232286555' on 192.168.199.91:8250 
	at com.cloud.agent.manager.ClusteredAgentManagerImpl.connectToPeer(ClusteredAgentManagerImpl.java:519)
	......
	at java.lang.Thread.run(Thread.java:748)
2020-07-16 10:48:10,642 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) Seq 1-3124935191441702917: Unable to forward null
2020-07-16 10:48:10,642 WARN  [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) Resource [Host:1] is unreachable: Host 1: Unable to reach the peer that the agent is connected
2020-07-16 10:48:10,642 WARN  [c.c.r.ResourceManagerImpl] (StatsCollector-5:ctx-b39a4359) (logid:27c99646) Unable to obtain host 1 statistics. 
2020-07-16

192.168.199.91 IP 登录UI 创建资源域

可以成功创建资源域，但是创建完成后只有192.168.199.91 访问UI时可以看到正常二级存储容量和查看系统VM 控制台。

且两个管理节点的日志文件依旧在不停报错

192.168.199.91

2020-07-16 11:40:32,613 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-37-thread-1:null) (logid:) A client/agent attempting connection from address=192.168.199.92 has presented these certificate(s):
Certificate [1] :
 Serial: 440e1b4702d33119
  Not Before:Wed Jul 15 23:31:03 CST 2020
  Not After:Sat Jul 09 11:31:03 CST 2050
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=manager2
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:[[2, manager2]]
Certificate [2] :
 Serial: 21ba59eaf4005763
  Not Before:Wed Jul 15 23:29:11 CST 2020
  Not After:Sat Jul 09 11:29:11 CST 2050
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=ca.cloudstack.apache.org
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:null
2020-07-16 11:40:32,615 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-37-thread-1:null) (logid:) Certificate ownership verification failed for client: 192.168.199.92
2020-07-16 11:40:32,618 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/192.168.199.91:8250, remote address=/192.168.199.92:45596.

192.168.199.92

2020-07-16 11:41:27,478 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-4:ctx-a07d9aa9) (logid:2603e149) Seq 1-2784350469621809162: Unable to forward null
2020-07-16 11:41:27,536 ERROR [c.c.u.n.Link] (StatsCollector-4:ctx-a07d9aa9) (logid:2603e149) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/192.168.199.92:45634, remote address=/192.168.199.91:8250. The client may have invalid ca-certificates.
2020-07-16 11:41:27,537 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-4:ctx-a07d9aa9) (logid:2603e149) Unable to connect to peer management server: 3232286555, ip: 192.168.199.91 due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '3232286555' on 192.168.199.91:8250 
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '3232286555' on 192.168.199.91:8250 
	at com.cloud.agent.manager.ClusteredAgentManagerImpl.connectToPeer(ClusteredAgentManagerImpl.java:519)
	......
	at java.lang.Thread.run(Thread.java:748)

因此 可能需要一个 负载均衡 IP 当作 host 的值。但好像也没成功

安装CloudStack时设置HAProxy。

manager1 设置

cloudstack-setup-databases cloud:123456@agent --deploy-as=root:123456
cloudstack-setup-management

manager2 设置

cloudstack-setup-databases cloud:123456@agent
cloudstack-setup-management

注，多个管理节点时数据库的配置
max_connections 等于 350 * 管理节点数量

max_connections=700

agent节点 cloudstack-agent设置同单个管理节点一致

HAProxy 设置

安装

yum -y install haproxy

设置日志文件使得 haproxy有日志记录。

编辑haproxy配置文件

vi /etc/haproxy/haproxy.cfg

global
    log         127.0.0.1 local2

编辑 rsyslog 配置文件 取消下列行注释

vi /etc/rsyslog.conf 

$ModLoad imudp
$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

并在末尾加上

local2.*                       /var/log/haproxy.log

编辑 rsyslog 文件

 vi /etc/sysconfig/rsyslog

SYSLOGD_OPTIONS="-r"

重启日志文件和haproxy

systemctl restart rsyslog
systemctl restart haproxy

HAProxy负载均衡设置

在defaults设置中加上 cookie 设置达到持久保持状态。

defaults
	......
    cookie cloud_cook  insert  nocache

cloud_cook 表示haproxy将在浏览器之上设置的cookie名，insert nocache表示cookie的设置方式。

frontend cloud_source_80
    bind *:80
    bind *:443
    use_backend cloud_dec_8080

backend cloud_dec_8080
    balance     roundrobin
    server manager1_8080 192.168.199.91:8080 cookie manager1 check inter 2000 rise 2 fall 5
    server manager2_8080 192.168.199.92:8080 cookie manager2 check inter 2000 rise 2 fall 5


frontend cloud_source_8250
    bind *:8250
	mode tcp
    use_backend cloud_dec_8250

backend cloud_dec_8250
    balance     roundrobin
	mode tcp
    server manager1_8250 192.168.199.91:8250 cookie manager1 check inter 2000 rise 2 fall 5
    server manager2_8250 192.168.199.92:8250 cookie manager2 check inter 2000 rise 2 fall 5


frontend cloud_source_8096
    bind *:8096
    use_backend cloud_dec_8096

backend cloud_dec_8096
    balance     roundrobin
    server manager1_8096 192.168.199.91:8096 check inter 2000 rise 2 fall 5
    server manager2_8096 192.168.199.92:8096 check inter 2000 rise 2 fall 5


listen stats
    bind :32700
    stats enable
    stats refresh 60s
    stats uri /
    stats hide-version
    stats auth admin:123456

cookie manager1表示haproxy 转发到192.168.199.91 时，cloud_cook的值为manager1。带有此cookie，发出的请求都会转发到192.168.199.91。这就保证了会话保持。
应该是这样。。。

重新启动改 haproxy

systemctl restart haproxy

访问 192.168.199.90 即可进入CloudStack的UI界面
修改 全局设置 host 值为 192.168.199.90

但是，不知什么原因，基本上添加主机一步会失败，不是因计算节点配置 Libvirtd失败，就是SSL问题失败。

2020-07-15 10:49:37,498 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-23-thread-1:null) (logid:) Certificate ownership verification failed for client: 192.168.199.90
2020-07-15 10:49:37,500 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/192.168.199.91:8250, remote address=/192.168.199.90:50178.

2020-07-15 10:49:52,770 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-24-thread-1:null) (logid:) Certificate ownership verification failed for client: 192.168.199.90
2020-07-15 10:49:52,771 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/192.168.199.92:8250, remote address=/192.168.199.90:45998.

似乎出现Libvirtd失败是因为网络问题。VMwareWorkstations虚拟机需要重置虚拟网络编辑器才可重新开始测试，不然就会出现Libvirtd启动失败

SSL 就真的解决不了了。。。

计算节点日志文件

2020-07-15 17:44:57,972 INFO  [cloud.agent.Agent] (main:null) (logid:) Connecting to host:192.168.199.90
2020-07-15 17:44:57,972 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.199.90:8250
2020-07-15 17:44:57,974 INFO  [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2020-07-15 17:44:58,061 ERROR [utils.nio.Link] (main:null) (logid:) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/192.168.199.93:38458, remote address=/192.168.199.90:8250. The client may have invalid ca-certificates.
2020-07-15 17:44:58,061 ERROR [utils.nio.NioClient] (main:null) (logid:) SSL Handshake failed while connecting to host: 192.168.199.90 port: 8250
2020-07-15 17:44:58,061 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.199.90 port: 8250
	at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
	at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
	at com.cloud.agent.Agent.start(Agent.java:293)
	at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
	at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
	at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
	at com.cloud.agent.AgentShell.start(AgentShell.java:512)
	at com.cloud.agent.AgentShell.main(AgentShell.java:547)
2020-07-15 17:44:58,062 INFO  [utils.exception.CSExceptionErrorCode] (main:null) (logid:) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2020-07-15 17:44:58,062 WARN  [cloud.agent.Agent] (main:null) (logid:) NIO Connection Exception  com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.199.90 port: 8250
2020-07-15 17:44:58,062 INFO  [cloud.agent.Agent] (main:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again...

但是又不想辛辛苦苦设置大半天，一点用都没有，因此，先记录到此。等以后有机会明白为什么时再来修改

本文地址：https://blog.csdn.net/dandanfengyun/article/details/107335916