一、搭建高可用OpenStack（Queen版）集群之部署Nova控制节点集群
一、简介
1、概述
使用Openstack计算服务来托管和管理云计算系统。Openstack计算服务是基础设施即服务(IaaS)系统的主要部分，模块主要由python实现。
Openstack计算组件请求Openstack Identitiy服务进行认证；请求Openstack Image服务提供磁盘镜像；为Openstack dashboard提供用户和管理员接口。磁盘镜像访问限制在项目与用户上；配额以每个项目进行设定。Openstack组件可以在标准硬件上水平大规模扩展，并且下载磁盘镜像启动虚拟机实例。
2、Openstack计算服务由下列组件所构成
（1） nova-api 服务：
接收和响应来自最终用户的计算API请求。此服务支持Openstack计算服务API，例如启动一个实例。
（2）nova-api-metadata 服务
接受来自虚拟机发送的元数据请求。
（3）nova-compute 服务
一个持续工作的守护进程，通过Hypervior的API来创建和销毁虚拟机实例，例如：
XenServer 的 XenAPI
KVM或QEMU的libvirt
过程是蛮复杂的。最为基本的，守护进程同意了来自队列的动作请求，转换为一系列的系统命令如启动一个KVM实例，然后到数据库中更新它的状态。
（4）nova-scheduler 服务
拿到一个来自队列请求虚拟机实例，然后决定那台物理计算节点来运行它。简单的说，该服务通过算法来决定在哪个计算节点来启动虚拟机。
（5）nova-cert 模块
Nova-network worker守护进程
从队列中接受网络任务，并操作网络。执行任务。例如创建网桥的接口或者改变iptables的规则。该功能应该被网络服务neutron取代。
Nova-consoleauth 守护进程
授权控制台代理所提供的用户令牌。
Nova-novncproxy 守护进程
提供一个代理，用于访问正在运行的实例，通过VNC协议，支持基于浏览器的novnc客户端。
队列
一个守护进程间传递消息的组件。前面部署的rabbitmq-server
SQL数据库
存储构建时和运行时的状态，为云基础设施，包括有：
可用实例类型
使用中的实例
可用网络
项目
理论上，OpenStack计算可以支持任何和SQL-Alchemy所支持的后端数据库，通常使用SQLite3来做测试可开发工作，MySQL和PostgreSQL 作生产环境。
二、部署Nova控制节点集群
1、创建nova相关数据库
在任意控制节点创建数据库，后台数据自动同步
nova服务含4个数据库，统一授权到nova用户；
placement主要涉及资源统筹，较常用的api接口是获取备选资源与claim资源等
mysql -u root -p
CREATE DATABASE nova_api;
CREATE DATABASE nova;
CREATE DATABASE nova_cell0;
CREATE DATABASE nova_placement;
GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'localhost' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'localhost' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON nova_placement.* TO 'nova'@'localhost' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON nova_placement.* TO 'nova'@'%' IDENTIFIED BY '123456';
flush privileges;
exit;
2、创建nova/placement-api
在任意控制节点操作
调用nova相关服务需要认证信息，加载环境变量脚本即可
. admin-openrc
1、创建nova/plcement用户
service项目已在glance章节创建；
nova/placement用户在”default” domain中
[root@controller01 ~]# openstack user create --domain default --password=nova_pass nova
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| domain_id | default |
| enabled | True |
| id | 2edf2fc7f9404e03bb33495957987617 |
| name | nova |
| options | {} |
| password_expires_at | None |
+---------------------+----------------------------------+
[root@controller01 ~]# openstack user create --domain default --password=placement_pass placement
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| domain_id | default |
| enabled | True |
| id
2、nova/placement赋权
为nova/placement用户赋予admin权限（没有返回值）
openstack role add --project service --user nova admin
openstack role add --project service --user placement admin
3、创建nova/placement服务实体
nova服务实体类型”compute”；
placement服务实体类型”placement”
[root@controller01 ~]# openstack service create --name nova --description "OpenStack Compute" compute
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Compute |
| enabled | True |
| id | 2160b3d55a8c42deb0e53d88f9a60893 |
| name | nova |
| type | compute |
+-------------+----------------------------------+
[root@controller01 ~]#"Placement API" placement
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Placement API |
| enabled | True |
| id
4、创建nova/placement-api
注意
- --region与初始化admin用户时生成的region一致；
- api地址统一采用vip，如果public/internal/admin分别使用不同的vip，请注意区分；
- nova-api 服务类型为compute，placement-api服务类型为placement；
# nova public api
openstack endpoint create --region RegionTest compute public http://controller:8774/v2.1
# nova internal api
openstack endpoint create --region RegionTest compute internal http://controller:8774/v2.1
# nova admin api
openstack endpoint create --region RegionTest compute admin http://controller:8774/v2.1
# placement public api
openstack endpoint create --region RegionTest placement public http://controller:8778
# placement internal api
openstack endpoint create --region RegionTest placement internal http://controller:8778
# placement admin api
openstack endpoint create --region RegionTest placement admin http://controller:8778
3、安装nova
在全部控制节点安装nova相关服务
yum install openstack-nova-api openstack-nova-conductor \
openstack-nova-console openstack-nova-novncproxy \
openstack-nova-scheduler openstack-nova-placement-api -y
4、配置nova.conf
在全部控制节点操作
注意：
- ”my_ip”参数，根据节点修改；
- nova.conf文件的权限：root:nova
cp -rp /etc/nova/nova.conf{,.bak}
egrep -v "^$|^#"[DEFAULT]
my_ip=10.20.9.189
use_neutron=true
firewall_driver=nova.virt.firewall.NoopFirewallDriver
enabled_apis=osapi_compute,metadata
osapi_compute_listen=$my_ip
osapi_compute_listen_port=8774
metadata_listen=$my_ip
metadata_listen_port=8775
# 前端采用haproxy时，服务连接rabbitmq会出现连接超时重连的情况，可通过各服务与rabbitmq的日志查看；
# transport_url=rabbit://openstack:openstack@controller:5673
# rabbitmq本身具备集群机制，官方文档建议直接连接rabbitmq集群；但采用此方式时服务启动有时会报错，原因不明；如果没有此现象，强烈建议连接rabbitmq直接对接集群而非通过前端haproxy
transport_url=rabbit://openstack:openstack@controller01:5672,openstack:openstack@controller02:5672,openstack:openstack@controller03:5672
[api]
auth_strategy=keystone
[api_database]
connection=mysql+pymysql://nova:123456@controller/nova_api
[barbican]
[cache]
backend=oslo_cache.memcache_pool
enabled=True
memcache_servers=controller01:11211,controller02:11211,controller03:11211connection = mysql+pymysql://nova:123456@controller/nova
[devices]
[ephemeral_storage_encryption]
[filter_scheduler]
[glance]
api_servers = http://controller:9292
[guestfs]
[healthcheck]
[hyperv]
[ironic]
[key_manager]
[keystone]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller01:11211,controller02:11211,controller03:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = nova_pass
[libvirt]
[matchmaker_redis]
[metrics]
[mks]
[neutron]
[notifications]
[osapi_v21]
[oslo_concurrency]
lock_path=/var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[pci]
[placement]
region_name = RegionTest
project_domain_name = Default
project_name = service
auth_type = password
user_domain_name = Default
auth_url = http://controller:35357/v3
username = placement
password = placement_pass
[quota]
[rdp]
[remote_debug]
[scheduler]
[serial_console]
[service_user]
[spice]
[upgrade_levels]
[vault]
[vendordata_dynamic_auth]
[vmware]
[vnc]
enabled=true
server_listen=$my_ip
server_proxyclient_address=$my_ip
novncproxy_base_url=http://$my_ip:6080/vnc_auto.html
novncproxy_host=$my_ip
novncproxy_port=6080
[workarounds]
[wsgi]
[xenserver]
[xvp]
同步配置完毕，更改IP
5、配置00-nova-placement-api.conf
在全部控制节点操作
注意根据不同节点修改监听地址
sed -i "s/Listen\ 8778/Listen\ 10.20.9.189:8778/g" /etc/httpd/conf.d/00-nova-placement-api.conf
sed -i "s/*:8778/10.20.9.189:8778/g" /etc/httpd/conf.d/00-nova-placement-api.conf
cp -rp /etc/httpd/conf.d/00-nova-placement-api.conf{,.bak}
echo "
#Placement API
<Directory /usr/bin>
<IfVersion >= 2.4>
Require all granted
</IfVersion>
<IfVersion < 2.4>
Order allow,deny
Allow from all
</IfVersion>
</Directory>
" >> /etc/httpd/conf.d/00-nova-placement-api.conf同步配置完毕，更改IP
#02
sed -i "s/10.20.9.189/10.20.9.190/g" /etc/httpd/conf.d/00-nova-placement-api.conf
#03
sed -i "s/10.20.9.189/10.20.9.45/g" /etc/httpd/conf.d/00-nova-placement-api.conf
重启httpd服务，启动placement-api监听端口
systemctl restart httpd
6、同步nova相关数据库
1、同步nova相关数据库
# 同步nova-api数据库
su -s /bin/sh -c "nova-manage api_db sync" nova
# 注册cell0数据库
su -s /bin/sh -c "nova-manage cell_v2 map_cell0" nova
# 创建cell1 cell (返回55c43661-28f2-4efa-a3d0-12d4b2769fa3)
su -s /bin/sh -c "nova-manage cell_v2 create_cell --name=cell1 --verbose" nova
# 同步nova数据库；
# 忽略”deprecated”信息
su -s /bin/sh -c "nova-manage db sync"
补充：此版本在向数据库同步导入数据表时，报错：
/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
exception.NotSupportedWarning
解决参考链接：https://cloud.tencent.com/info/fe786243c2657ac2850618075e00b6d9.html
根据报错信息找到对应的文件，注释第325、329、330、331、332和333行的配置。
sed -n '325,333p' /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py
结果如下
# if not_supported:
# # would like to raise ValueError here, but there are just
# # too many unrecognized (obsolete?) configuration options
# # coming in from projects
# warnings.warn(
# "Configuration option(s) %r not supported" %
# sorted(not_supported),
# exception.NotSupportedWarning
# )
7、启动服务
在全部控制节点操作
开机自启
systemctl enable openstack-nova-api.service \
openstack-nova-consoleauth.service \
openstack-nova-scheduler.service \
openstack-nova-conductor.service \
openstack-nova-novncproxy.service
启动
systemctl restart openstack-nova-api.service
systemctl restart openstack-nova-consoleauth.service
systemctl restart openstack-nova-scheduler.service
systemctl restart openstack-nova-conductor.service
systemctl restart openstack-nova-novncproxy.service
查看状态
systemctl status openstack-nova-api.service \
openstack-nova-consoleauth.service \
openstack-nova-scheduler.service \
openstack-nova-conductor.service \
openstack-nova-novncproxy.service
查看端口
netstat -tunlp | egrep '8774|8775|8778|6080'
8、验证
在任意控制节点操作
. admin-openrc
列出各服务组件，查看状态
使用命令nova service-list 或者openstack compute service list
[root@controller01 ~]# openstack compute service list
+-----+------------------+--------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+-----+------------------+--------------+----------+---------+-------+----------------------------+
| 85 | nova-consoleauth | controller01 | internal | enabled | up | 2018-10-15T09:41:58.000000 |
| 88 | nova-consoleauth | controller03 | internal | enabled | up | 2018-10-15T09:42:00.000000 |
| 91 | nova-consoleauth | controller02 | internal | enabled | up | 2018-10-15T09:42:00.000000 |
| 94 | nova-scheduler | controller01 | internal | enabled | up | 2018-10-15T09:42:02.000000 |
| 97 | nova-scheduler | controller03 | internal | enabled | up | 2018-10-15T09:42:04.000000 |
| 100 | nova-scheduler | controller02 | internal | enabled | up | 2018-10-15T09:42:04.000000 |
| 103 | nova-conductor | controller01 | internal | enabled | up | 2018-10-15T09:42:05.000000 |
| 115 | nova-conductor | controller03 | internal | enabled | up | 2018-10-15T09:41:58.000000 |
| 124 | nova-conductor | controller02 | internal | enabled | up | 2018-10-15T09:41:58.000000 |
+-----+------------------+--------------+----------+---------+-------+----------------------------+
展示api端点
[root@controller01 ~]# openstack catalog list
+-----------+-----------+-----------------------------------------+
| Name | Type | Endpoints |
+-----------+-----------+-----------------------------------------+
| nova | compute | RegionTest |
| | | public: http://controller:8774/v2.1 |
| | | RegionTest |
| | | internal: http://controller:8774/v2.1 |
| | | RegionTest |
| | | admin: http://controller:8774/v2.1 |
| | | |
| keystone | identity | RegionTest |
| | | internal: http://controller:5000/v3/ |
| | | RegionTest |
| | | public: http://controller:5000/v3/ |
| | | RegionTest |
| | | admin: http://controller:35357/v3/ |
| | | |
| glance | image | RegionTest |
| | | internal: http://controller:9292 |
| | | RegionTest |
| | | admin: http://controller:9292 |
| | | RegionTest |
| | | public: http://controller:9292 |
| | | |
| placement | placement | RegionTest |
| | | public: http://controller:8778 |
| | | RegionTest |
| | | admin: http://controller:8778 |
| | | RegionTest |
| | | internal: http://controller:8778 |
检查cell与placement api运行正常
[root@controller01 ~]# nova-status upgrade check
+--------------------------------+
| Upgrade Check Results |
+--------------------------------+
| Check: Cells v2 |
| Result: Success |
| Details: None |
+--------------------------------+
| Check: Placement API |
| Result: Success |
| Details: None |
+--------------------------------+
| Check: Resource Providers |
| Result: Success |
| Details: None |
+--------------------------------+
| Check: Ironic Flavor Migration |
| Result: Success |
| Details: None |
+--------------------------------+
| Check: API Service Version |
| Result: Success |
| Details: None |
+--------------------------------+
9、设置pcs资源
在任意控制节点操作
1、添加资源openstack-nova-api，openstack-nova-consoleauth，openstack-nova-scheduler，openstack-nova-conductor与openstack-nova-novncproxy
pcs resource create openstack-nova-api systemd:openstack-nova-api --clone interleave=true
pcs resource create openstack-nova-consoleauth systemd:openstack-nova-consoleauth --clone interleave=true
pcs resource create openstack-nova-scheduler systemd:openstack-nova-scheduler --clone interleave=true
pcs resource create openstack-nova-conductor systemd:openstack-nova-conductor --clone interleave=true
pcs resource create openstack-nova-novncproxy systemd:openstack-nova-novncproxy --clone interleave=true
经验证，建议openstack-nova-api，openstack-nova-consoleauth，openstack-nova-conductor与openstack-nova-novncproxy 等无状态服务以active/active模式运行；openstack-nova-scheduler等服务以active/passive模式运行
2、查看pcs资源
[root@controller01 ~]# pcs resource
vip (ocf::heartbeat:IPaddr2): Started controller01
Clone Set: lb-haproxy-clone [lb-haproxy]
Started: [ controller01 ]
Stopped: [ controller02 controller03 ]
Clone Set: openstack-keystone-clone [openstack-keystone]
Started: [ controller01 controller02 controller03 ]
Clone Set: openstack-glance-api-clone [openstack-glance-api]
Started: [ controller01 controller02 controller03 ]
Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
Started: [ controller01 controller02 controller03 ]
Clone Set: openstack-nova-api-clone [openstack-nova-api]
Started: [ controller01 controller02 controller03 ]
Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
Started: [ controller01 controller02 controller03 ]
Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
Started: [ controller01 controller02 controller03 ]
Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
Started: [ controller01 controller02 controller03 ]
Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
Started: [ controller01 controller02 controller03 ]