零 修订记录

一 摘要

本文介绍基于9台物理机部署openstack ussuri版本 以及与ceph nautils集成。

二环境信息

（一）服务器信息

（二）交换机信息

两台相同配置的交换机配置堆叠。

(三)软件信息

2.3.1 操作系统

[root@localhost ~]# cat /etc/centos-release
CentOS Linux release 8.1.1911 (Core)
[root@localhost ~]# uname -a
Linux localhost.localdomain 4.18.0-147.el8.x86_64 #1 SMP Wed Dec 4 21:51:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]#

2.3.2 docker

2.3.3 kolla

2.3.4 kolla-ansible

三实施

（一）部署规划

3.1.1 部署网络规划

两台交互及做堆叠，服务器上有两张万兆网卡，每张网卡有两个port，交叉制作bond0,bond1;
10.3.140.0/24 bond0 用于物理机地址及api管理
10.3.141.0/24 专用于存储集群交换

10.3.142.0/24-10.3.149.0/24 用于虚拟机；

3.1.2 部署节点功能规划

（二）部署ceph 集群

ceph 集群部署请参照linux （centos 7.6）生产环境基于三台物理机 安装 ceph 集群以及集成openstack

(三) 部署openstack ussuri集群

我这里是离线部署，关于如何下载openstack ussuri包，及相关依赖
请参考openstack ussuri版本基于内网三台物理机集群kolla-ansible部署及与ceph 集群 集成

这里主要下载两类
1.是镜像
2.是部署节点安装时需要的各种依赖

3.3.1 部署前准备（控制节点和计算节点都要）

3.3.1.1 关闭防火墙以及selinux

[dev@10-3-170-32 base]$ ansible-playbook closefirewalldandselinux.yml

需要重启机器

3.3.1.2 配置dns以及修改机器名

dns 服务器上配置参考下表

修改机器名：

[root@localhost ~]# hostnamectl set-hostname procontroller01.pro.kxdigit.com
[root@localhost ~]#

以上以一台机器为例，9台机器均需设置，其中存储节点在ceph 安装时已设置，故此处只需设置6台即可。

修改服务器dns 解析地址

[dev@10-3-170-32 base]$ ansible-playbook modifydns.yml

修改ssh 配置文件 /etc/ssh/sshd_config，取消登录时调用dns,重启ssh

#UseDNS yes
UseDNS no

  systemctl restart sshd

3.3.1.3 更新yum 源

更新操作系统yum,更新docker 源

[dev@10-3-170-32 base]$ ansible-playbook updateyum.yml

[dev@10-3-170-32 base]$ ansible-playbook updateansible.yml

[root@procontroller01 ~]# ll /etc/yum.repos.d
total 12
-rw-r--r-- 1 root root 501 Apr 29 14:55 CentOS-81.repo
-rw-r--r-- 1 root root  91 Apr 29 15:26 ansible2910forcentos8.repo
-rw-r--r-- 1 root root 104 Apr 29 14:55 docker1803forcentos8.repo
[root@procontroller01 ~]#

3.3.1.4 同步时间服务器

[dev@10-3-170-32 base]$ ansible-playbook modifychronyclient.yml

3.3.1.5 安装docker

控制节点 计算节点 逐台安装,以下一控制节点一为例只需
安装、修改配置文件、设置开机启动

我这里目前安装的是docker 18.03,三台节点都需要安装
我这里已经配置了docker yum 源，直接yum install docker-ce 即可安装，安装时可能会报错，主要是依赖包兼容问题。
安装docker 异常处理参考openstack ussuri 版本 all-in-one 离线部署

[root@procontroller01 ~]#  yum install docker-ce
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:01:16 ago on Thu Apr 29 14:56:36 2021.
Modular dependency problems:

若有该包兼容性错误，可以删除

 yum remove podman

docker 配置文件修改

新增 /etc/systemd/system/docker.service.d/kolla.conf

[root@procontroller01 ~]# cd /etc/systemd/system/

[root@procontroller01 system]# mkdir docker.service.d
[root@procontroller01 system]# cd docker.service.d/
[root@procontroller01 docker.service.d]# vi kolla.conf
[root@procontroller01 docker.service.d]#

文件内容如下

[Service]
MountFlags=shared

设置开机启动

[root@procontroller01 docker.service.d]# systemctl daemon-reload
[root@procontroller01 docker.service.d]# systemctl restart docker && systemctl enable docker && systemctl status docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─kolla.conf
   Active: active (running) since Thu 2021-04-29 15:01:14 CST; 126ms ago
     Docs: https://docs.docker.com
 Main PID: 36902 (dockerd)
    Tasks: 39 (limit: 32767)
   Memory: 55.6M
   CGroup: /system.slice/docker.service
           ├─36902 /usr/bin/dockerd
           └─36914 docker-containerd --config /var/run/docker/containerd/containerd.toml

Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.185105356+08:00" level=info msg="Graph>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.185371260+08:00" level=warning msg="Yo>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.185384710+08:00" level=warning msg="Yo>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.185806534+08:00" level=info msg="Loadi>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.272931021+08:00" level=info msg="Defau>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.331678817+08:00" level=info msg="Loadi>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.363650311+08:00" level=info msg="Docke>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.363785574+08:00" level=info msg="Daemo>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com dockerd[36902]: time="2021-04-29T15:01:14.369294430+08:00" level=info msg="API l>
Apr 29 15:01:14 procontroller01.pro.kxdigit.com systemd[1]: Started Docker Application Container Engine.

3.3.1.6 安装dpip

centos8.1 自带python3，默然安装了pip3.6 ，我们只需要做下链接即可。

3.3.2.2.1 制作pip链接

[root@procontroller01 ~]# whereis pip3.6
pip3: /usr/bin/pip3 /usr/bin/pip3.6 /usr/share/man/man1/pip3.1.gz
[root@procontroller01 ~]# ln -s /usr/bin/pip3.6 /usr/bin/pip
[root@procontroller01 ~]# pip -V
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)
[root@procontroller01 ~]#

3.3.1.7 禁用libvirt(仅计算节点执行)

[root@procompute03 ~]# systemctl stop libvirtd.service && systemctl disable libvirtd.service && systemctl status libvirtd.service
Removed /etc/systemd/system/multi-user.target.wants/libvirtd.service.
Removed /etc/systemd/system/sockets.target.wants/virtlogd.socket.
Removed /etc/systemd/system/sockets.target.wants/virtlockd.socket.
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2021-04-29 16:40:50 CST; 108ms ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 5342 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/libvirtd.service
           ├─5814 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/>
           └─5815 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/>

Apr 29 14:45:01 procompute03.pro.kxdigit.com dnsmasq-dhcp[5814]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Apr 29 14:45:01 procompute03.pro.kxdigit.com dnsmasq-dhcp[5814]: DHCP, sockets bound exclusively to interface virbr0
Apr 29 14:45:01 procompute03.pro.kxdigit.com dnsmasq[5814]: no servers found in /etc/resolv.conf, will retry
Apr 29 14:45:01 procompute03.pro.kxdigit.com dnsmasq[5814]: read /etc/hosts - 2 addresses
Apr 29 14:45:01 procompute03.pro.kxdigit.com dnsmasq[5814]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Apr 29 14:45:01 procompute03.pro.kxdigit.com dnsmasq-dhcp[5814]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Apr 29 14:49:16 procompute03.pro.kxdigit.com dnsmasq[5814]: reading /etc/resolv.conf
Apr 29 14:49:16 procompute03.pro.kxdigit.com dnsmasq[5814]: using nameserver 10.3.157.201#53
Apr 29 16:40:50 procompute03.pro.kxdigit.com systemd[1]: Stopping Virtualization daemon.

3.3.2 安装部署服务(部署服务器)

3.3.2.1 安装ansible

3.3.2.1.1 安装ansible

[root@procontroller01 ~]# yum install ansible
Failed to set locale, defaulting to C.UTF-8
ansible                                        

[root@procontroller01 ~]# ansible --version
ansible 2.9.10
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
[root@procontroller01 ~]#

3.3.2.1.2 修改ansible 配置文件

对ansible /etc/ansible/ansible.cfg 配置文件做了优化
先备份原文件：cp /etc/ansible/ansible.cfg /etc/ansible/ansible.cfg.bak.orig

新增如下内容：

[defaults]
inventory  = $HOME/ansible/hosts
host_key_checking=False
pipelining=True
forks=10

3.3.2.2 安装基础依赖

centos8 使用dnf 安装rpm 包，安装python3-devel libffi-devel gcc openssl-devel python3-libselinux git vim bash-completion net-tools 基础依赖包

3.3.2.2.1 安装

[root@procontroller01 ussuri]# cd dependencies/
[root@procontroller01 dependencies]# ll
total 31224
-rw-r--r-- 1 root root   280084 Jun 17  2020 bash-completion-2.7-5.el8.noarch.rpm
-rw-r--r-- 1 root root 24564532 Jun 17  2020 gcc-8.3.1-4.5.el8.x86_64.rpm
-rw-r--r-- 1 root root   190956 Jun 17  2020 git-2.18.2-2.el8_1.x86_64.rpm
-rw-r--r-- 1 root root    29396 Jun 17  2020 libffi-devel-3.1-21.el8.i686.rpm
-rw-r--r-- 1 root root    29376 Jun 17  2020 libffi-devel-3.1-21.el8.x86_64.rpm
-rw-r--r-- 1 root root   330916 Jun 17  2020 net-tools-2.0-0.51.20160912git.el8.x86_64.rpm
-rw-r--r-- 1 root root  2395376 Jun 17  2020 openssl-devel-1.1.1c-2.el8_1.1.i686.rpm
-rw-r--r-- 1 root root  2395344 Jun 17  2020 openssl-devel-1.1.1c-2.el8_1.1.x86_64.rpm
-rw-r--r-- 1 root root   290084 Jun 17  2020 python3-libselinux-2.9-2.1.el8.x86_64.rpm
-rw-r--r-- 1 root root    16570 Jun 17  2020 python36-devel-3.6.8-2.module_el8.1.0+245+c39af44f.x86_64.rpm
-rw-r--r-- 1 root root  1427224 Jun 17  2020 vim-enhanced-8.0.1763-13.el8.x86_64.rpm
[root@procontroller01 dependencies]# dnf install python3-devel libffi-devel gcc openssl-devel python3-libselinux git vim bash-completion net-tools
Failed to set locale, defaulting to C.UTF-8

3.3.2.3 安装kolla （部署节点）

[root@procontroller01 ussuri]# unzip kolla-10.1.0.zip
Archive:  kolla-10.1.0.zip
fbea3bf26d93e6fd784dbd9967659f549df3ec2d

3.3.2.3.1离线安装

[root@procontroller01 ussuri]# cd kolla-10.1.0/
[root@procontroller01 kolla-10.1.0]# git init
Initialized empty Git repository in /root/ussuri/kolla-10.1.0/.git/
[root@procontroller01 kolla-10.1.0]#

安装依赖包

[root@procontroller01 ~]# pip install --no-index --find-links=/root/software/ussuri/kollapip -r /root/software/ussuri/kolla-10.1.0/requirements.txt
WARNING: Running pip 

安装kolla

[root@procontroller01 ~]# pip install /root/software/ussuri/kolla-10.1.0/
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Processing ./software/ussuri/kolla-10.1.0

验证

[root@procontroller01 ~]# kolla-build --version
0.0.0
[root@procontroller01 ~]#

3.3.2.4 安装kolla-ansible (部署节点)

本文安装的是kolla-ansible 10.1.0 版本

3.3.2.4.1离线安装

解压

[root@procontroller01 ussuri]# unzip kolla-ansible-10.1.0.zip
Archive:  kolla-ansible-10.1.0.zip
6bba8cc52af3a26678da48129856f80c21eb8e38

git 初始化

[root@procontroller01 ussuri]# cd kolla-ansible-10.1.0/
[root@procontroller01 kolla-ansible-10.1.0]# git init
Initialized empty Git repository in /root/software/ussuri/kolla-ansible-10.1.0/.git/
[root@procontroller01 kolla-ansible-10.1.0]#

安装kolla-ansible 依赖包

[root@procontroller01 kolla-ansible-10.1.0]# pip install --no-index --find-links=/root/software/ussuri//kollaansiblepip -r /root/software/ussuri/kolla-ansible-10.1.0/requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/local/lib/python3.6/site-packages (from -r /root/software/ussuri/kolla-ansible-10.1.0/requirements.txt (line 1))

安装kolla-ansible

[root@procontroller01 kolla-ansible-10.1.0]#  pip install /root/software/ussuri/kolla-ansible-10.1.0
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Processing /root/software/ussuri/kolla-ansible-10.1.0

验证

[root@procontroller01 kolla-ansible-10.1.0]# kolla-ansible -h
Usage: /usr/local/bin/kolla-ansible COMMAND [options]

Options:
    --inventory, -i <inventory_path>   Specify path to ansible inventory file
    --playbook, -p <playbook_path>     Specify path to ansible playbook file
    --configdir <config_path>          Specify path to directory with globals.yml
    --key -k <key_path>                Specify path to ansible vault keyfile
    --help, -h                         Show this usage information

3.3.2.4.2 新增kolla-ansilbe 配置文件

[root@procontroller01 kolla-ansible-10.1.0]# mkdir -p /etc/kolla
[root@procontroller01 kolla-ansible-10.1.0]# cp -r /root/software/ussuri/kolla-ansible-10.1.0/etc/kolla/* /etc/kolla
[root@procontroller01 kolla-ansible-10.1.0]# ll /etc/kolla/
total 36
-rw-r--r-- 1 root root 25509 Apr 29 15:55 globals.yml
-rw-r--r-- 1 root root  5037 Apr 29 15:55 passwords.yml
[root@procontroller01 kolla-ansible-10.1.0]#

3.3.2.4.2 新增ansilbe 配置文件

[root@procontroller01 kolla-ansible-10.1.0]# mkdir /root/ansible
[root@procontroller01 kolla-ansible-10.1.0]# cp /root/software/ussuri/kolla-ansible-10.1.0/ansible/inventory/* /root/ansible/
[root@procontroller01 kolla-ansible-10.1.0]# ll /root/ansible/
total 24
-rw-r--r-- 1 root root  9584 Apr 29 15:56 all-in-one
-rw-r--r-- 1 root root 10058 Apr 29 15:56 multinode
[root@procontroller01 kolla-ansible-10.1.0]#

3.3.2.5 部署节点免密登录到所有节点

部署节点能免密登录到所有控制节点和计算节点，
下面仅示例登录到控制节点1，剩下六台 参考方法

[root@procontroller01 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Fol6UrRlbAcOreLvCDG3cknIk3+C8a0KeF9wIDuD4sw root@procontroller01.pro.kxdigit.com
The key's randomart image is:
+---[RSA 3072]----+
|      oo+.       |
|     . B+..      |
|  . . +o+.       |
| o =.+.  .       |
|o %.*.o S        |
|*  &.O .         |
|oE= B.+          |
| o = *.          |
|  ..+..          |
+----[SHA256]-----+
[root@procontroller01 ~]# ssh-copy-id root@10.3.140.11
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.3.140.11 (10.3.140.11)' can't be established.
ECDSA key fingerprint is SHA256:j5XQyrGFUqdRSnbrryQo7oD+SjATitiH5MC7wJTW3EQ.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

3.3.3 kolla ansible 部署openstack准备工作

下面上传的是train 版本镜像，ussuri 版本上传方法类似，只是上传的内容以及镜像tag 不同。

这里主要是配置文件的初步修改，镜像上传到内网镜像中心,
我这里内网有个docker 仓库，而且是https方式
https://registry.kxdigit.com

非https 方式，docker 需要配置。

3.3.3.1 镜像上传到内网镜像中心

3.3.3.1.1 上传到内网

部署节点新增目录

[root@procontroller01 software]# mkdir -p ./openstack/train/images/kolla
[root@procontroller01 software]# cd openstack/train/images/kolla/
[root@procontroller01 kolla]# pwd
/root/software/openstack/train/images/kolla
[root@procontroller01 kolla]#

openstak 镜像导入内网

此步骤略
我这边镜像到上传到部署节点,总共134个镜像

centos-source-zun-api.tar.gz                                                                   100%  337MB 112.2MB/s   00:03
centos-source-zun-compute.tar.gz                                                               100%  387MB  96.7MB/s   00:04
centos-source-zun-wsproxy.tar.gz                                                               100%  337MB 112.2MB/s   00:03
[dev@10-3-170-32 kolla]$ pwd
/home/dev/software/openstack/train/images/kolla
[dev@10-3-170-32 kolla]$

[dev@10-3-170-32 kolla]$ ls | grep gz |wc -l
134
[dev@10-3-170-32 kolla]$

3.3.3.1.2 解压镜像压缩包

解压脚本

#! /bin/sh

for i in ` ls -al /root/software/openstack/train/images/kolla/  | grep gz | awk '{print $9}' `;do gzip -d /root/software/openstack/train/images/kolla/$i; done

3.3.3.1.3 批量加载镜像

脚本

[root@procontroller01 scripts]# cat dockerload.sh
#!/bin/bash

  path=$1
  cd $path

  for filename in `ls`
  do
    echo $filename
    docker load < $filename
  done

[root@procontroller01 scripts]#

执行命令：

[root@procontroller01 scripts]# ./dockerload.sh /root/software/openstack/train/images/kolla/
centos-source-aodh-api.tar
174f56854903: Loading layer [==================================================>]  211.7MB/211.7MB
1f66661b2a8e: Loading layer [==================================================>]  84.99kB/84.99kB

检查，已全部上传到部署节点

[root@procontroller01 kolla]# docker images | grep train | wc -l
134
[root@procontroller01 kolla]#

[root@procontroller01 kolla]# docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
kolla/centos-source-nova-compute                        train               bb089a9b5fdf        4 weeks ago         2.08GB
kolla/centos-source-horizon                             train               a4b1597d0c85        4 weeks ago         1.19GB

3.3.3.1.4 上传到内网docker 仓库

[root@procontroller01 scripts]# for i in `docker images|grep -v registry | grep  train|awk '{print $1}'`;do docker image tag $i:train registry.kxdigit.com/$i:train;done
[root@procontroller01 scripts]#

经比对 上面命令少执行了一条

[root@procontroller01 kolla]# docker images | grep centos-source-glance-registry
kolla/centos-source-glance-registry                                          train               f59b22154362        4 weeks ago         1.02GB
[root@procontroller01 kolla]# docker images|grep centos-source-glance-registry | grep  train|awk '{print $1}'
kolla/centos-source-glance-registry
[root@procontroller01 kolla]# for i in `docker images|grep centos-source-glance-registry | grep  train|awk '{print $1}'`;do docker image tag $i:train registry.kxdigit.com/$i:train;done
[root@procontroller01 kolla]#

先登录私有仓库

[root@procontroller01 kolla]# docker login -u admin -p passwordxxx  registry.kxdigit.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

批量推送到私有仓库

for i in ` docker images | grep kxdigit |awk '{print $1}'`;do docker push $i;done

3.3.3.2 生成密码文件并修改登录密码（部署节点）

先使用kolla-genpwd 生成全套密码，然后为了前台便于登录，修改keystone_admin_password 密码 为Admin_PASS2021

[root@procontroller01 scripts]# cd /etc/kolla/
[root@procontroller01 kolla]# ll
总用量 36
-rw-r--r--. 1 root root 24901 4月  27 11:07 globals.yml
-rw-r--r--. 1 root root  5217 4月  27 11:07 passwords.yml
[root@procontroller01 kolla]# kolla-genpwd
[root@procontroller01 kolla]# ll
总用量 56
-rw-r--r--. 1 root root 24901 4月  27 11:07 globals.yml
-rw-r--r--. 1 root root 25916 4月  27 17:42 passwords.yml
[root@procontroller01 kolla]# vim passwords.yml
[root@procontroller01 kolla]#

keepalived_password: 02u493LqhiQ8RXp0Zux4bEeZs0sr0AbxfFjb6CvZ
keystone_admin_password: Admin_PASS2021
keystone_database_password: DjD2mvrS0sgii9Wriz8emSBAekdBNXjDhjwwaa98

3.3.3.3 修改multinode（部署节点）

主要修改点有：
[control]
[network]
[compute]
[monitoring]
[storage]
这条我这里暂时没改，先试试能不能用
[nova-compute-ironic:children] 配置改为compute

仅放了修改的地方

[root@procontroller01 ansible]# cat multinode
# These initial groups are the only groups required to be modified. The
# additional groups are for more control of the environment.
[control]
# These hostname must be resolvable from your deployment host
procontroller01.pro.kxdigit.com
procontroller02.pro.kxdigit.com
procontroller03.pro.kxdigit.com

# The above can also be specified as follows:
#control[01:03]     ansible_user=kolla

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
procontroller01.pro.kxdigit.com
procontroller02.pro.kxdigit.com
procontroller03.pro.kxdigit.com

[compute]
procompute01.pro.kxdigit.com
procompute02.pro.kxdigit.com
procompute03.pro.kxdigit.com

[monitoring]
procontroller02.pro.kxdigit.com

# When compute nodes and control nodes use different interfaces,
# you need to comment out "api_interface" and other interfaces from the globals.yml
# and specify like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1

[storage]

[deployment]
localhost       ansible_connection=local


[nova-compute-ironic:children]
#nova
compute

3.3.3.4 修改global.yml（部署节点）

此步暂略，后面与ceph 集成时也要修改该文件，整合到一起。

3.3.4 openstack与 ceph 集成

3.3.4.1 修改global.yml

[root@procontroller01 kolla]#  cat /etc/kolla/globals.yml |grep -v "^#" | grep -v ^$
---
kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "ussuri"
kolla_internal_vip_address: "10.3.140.10"
docker_registry: registry.kxdigit.com
network_interface: "bond0"
neutron_external_interface: "bond1"
neutron_plugin_agent: "openvswitch"
openstack_logging_debug: "True"
enable_openstack_core: "yes"
enable_haproxy: "yes"
enable_cinder: "yes"
enable_manila_backend_cephfs_native: "yes"
ceph_glance_keyring: "ceph.client.glance.keyring"
ceph_glance_user: "glance"
ceph_glance_pool_name: "images"
ceph_cinder_keyring: "ceph.client.cinder.keyring"
ceph_cinder_user: "cinder"
ceph_cinder_pool_name: "volumes"
ceph_cinder_backup_keyring: "ceph.client.cinder-backup.keyring"
ceph_cinder_backup_user: "cinder-backup"
ceph_cinder_backup_pool_name: "backups"
ceph_nova_keyring: "{{ ceph_cinder_keyring }}"
ceph_nova_user: "cinder"
ceph_nova_pool_name: "vms"
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
nova_backend_ceph: "yes"
nova_compute_virt_type: "qemu"
[root@procontroller01 kolla]#

3.3.4.2 ceph 状态检查

[cephadmin@proceph01 ~]$ ceph -s
  cluster:
    id:     ad0bf159-1b6f-472b-94de-83f713c339a3
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum proceph01,proceph02,proceph03 (age 2d)
    mgr: proceph01(active, since 2d), standbys: proceph02, proceph03
    osd: 18 osds: 18 up (since 2d), 18 in (since 2d)

  data:
    pools:   4 pools, 544 pgs
    objects: 4 objects, 76 B
    usage:   18 GiB used, 131 TiB / 131 TiB avail
    pgs:     544 active+clean

[cephadmin@proceph01 ~]$

[cephadmin@proceph01 ~]$ ceph health detail
HEALTH_OK
[cephadmin@proceph01 ~]$

[cephadmin@proceph01 ~]$ ceph osd tree
ID CLASS WEIGHT    TYPE NAME          STATUS REWEIGHT PRI-AFF
-1       130.98395 root default
-3        43.66132     host proceph01
 0   hdd   7.27689         osd.0          up  1.00000 1.00000
 1   hdd   7.27689         osd.1          up  1.00000 1.00000
 2   hdd   7.27689         osd.2          up  1.00000 1.00000
 3   hdd   7.27689         osd.3          up  1.00000 1.00000
 4   hdd   7.27689         osd.4          up  1.00000 1.00000
 5   hdd   7.27689         osd.5          up  1.00000 1.00000
-5        43.66132     host proceph02
 6   hdd   7.27689         osd.6          up  1.00000 1.00000
 7   hdd   7.27689         osd.7          up  1.00000 1.00000
 8   hdd   7.27689         osd.8          up  1.00000 1.00000
 9   hdd   7.27689         osd.9          up  1.00000 1.00000
10   hdd   7.27689         osd.10         up  1.00000 1.00000
11   hdd   7.27689         osd.11         up  1.00000 1.00000
-7        43.66132     host proceph03
12   hdd   7.27689         osd.12         up  1.00000 1.00000
13   hdd   7.27689         osd.13         up  1.00000 1.00000
14   hdd   7.27689         osd.14         up  1.00000 1.00000
15   hdd   7.27689         osd.15         up  1.00000 1.00000
16   hdd   7.27689         osd.16         up  1.00000 1.00000
17   hdd   7.27689         osd.17         up  1.00000 1.00000
[cephadmin@proceph01 ~]$

3.3.4.3 配置glance 使用ceph

3.3.4.3.1 生成 ceph.client.glance.keyring

在ceph 部署节点 生成 ceph.client.glance.keyring 文件

[cephadmin@proceph01 ~]$ ceph auth get-or-create client.glance | tee /etc/ceph/ceph.client.glance.keyring
[client.glance]
        key = AQCYS4VgKKk7MRAAQZkwE3ISG1J+jsN1AcpqRg==
[cephadmin@proceph01 ~]$ ll /etc/ceph/
total 16
-rw-------. 1 cephadmin cephadmin 151 Apr 23 18:01 ceph.client.admin.keyring
-rw-rw-r--  1 cephadmin cephadmin  64 Apr 27 19:03 ceph.client.glance.keyring
-rw-r--r--  1 root      root      308 Apr 25 16:57 ceph.conf
-rw-r--r--. 1 cephadmin cephadmin  92 Nov 24 03:33 rbdmap
-rw-------. 1 cephadmin cephadmin   0 Apr 23 17:58 tmpd8sfbW
[cephadmin@proceph01 ~]$

3.3.4.3.2 koll-ansible 部署节点 新建/etc/kolla/config/glance

[root@procontroller01 ~]# mkdir -p /etc/kolla/config/glance
[root@procontroller01 ~]#  cd /etc/kolla/config/glance/
[root@procontroller01 glance]# ll
总用量 0
[root@procontroller01 glance]# pwd
/etc/kolla/config/glance
[root@procontroller01 glance]#

3.3.4.3.3 ceph.client.glance.keyring 文件从ceph部署节点拷贝到openstack 部署节点该目录下/etc/kolla/config/glance

[cephadmin@proceph01 ceph]$ scp /etc/ceph/ceph.conf root@10.3.140.11:/etc/kolla/config/glance/
root@10.3.140.11's password:
ceph.conf                                                                                      100%  308   588.9KB/s   00:00
[cephadmin@proceph01 ceph]$

只保留如下内容

[global]
fsid = ad0bf159-1b6f-472b-94de-83f713c339a3
mon_host = 10.3.140.31,10.3.140.32,10.3.140.33

3.3.4.4 配置cinder 使用ceph

3.3.4.4.1 ceph 部署节点生成 ceph.client.cinder.keyring 和 ceph.client.cinder-backup.keyring 密钥文件

[cephadmin@proceph01 ceph]$ ceph auth get-or-create client.cinder |  tee /etc/ceph/ceph.client.cinder.keyring
[client.cinder]
        key = AQCrS4VgBJhtDhAA5X+b5N7GJMOK/4p12IwRBg==
[cephadmin@proceph01 ceph]$ ceph auth get-or-create client.cinder-backup | tee /etc/ceph/ceph.client.cinder-backup.keyring
[client.cinder-backup]
        key = AQC5S4VgmMBDIRAA6xnBwr04Ik0lmn0WIoD20Q==
[cephadmin@proceph01 ceph]$ ll /etc/ceph/
total 24
-rw-------. 1 cephadmin cephadmin 151 Apr 23 18:01 ceph.client.admin.keyring
-rw-rw-r--  1 cephadmin cephadmin  71 Apr 27 19:13 ceph.client.cinder-backup.keyring
-rw-rw-r--  1 cephadmin cephadmin  64 Apr 27 19:13 ceph.client.cinder.keyring
-rw-rw-r--  1 cephadmin cephadmin  64 Apr 27 19:03 ceph.client.glance.keyring
-rw-r--r--  1 root      root      308 Apr 25 16:57 ceph.conf
-rw-r--r--. 1 cephadmin cephadmin  92 Nov 24 03:33 rbdmap
-rw-------. 1 cephadmin cephadmin   0 Apr 23 17:58 tmpd8sfbW
[cephadmin@proceph01 ceph]$

3.3.4.4.2 kolla-ansible 部署节点新增cinder 配置文件目录

[root@procontroller01 glance]# mkdir -p /etc/kolla/config/cinder/cinder-volume/
[root@procontroller01 glance]# mkdir -p /etc/kolla/config/cinder/cinder-backup
[root@procontroller01 glance]# ll /etc/kolla/config/
总用量 0
drwxr-xr-x. 4 root root 48 4月  27 19:14 cinder
drwxr-xr-x. 2 root root 57 4月  27 19:14 glance
[root@procontroller01 glance]# ll /etc/kolla/config/cinder/
总用量 0
drwxr-xr-x. 2 root root 6 4月  27 19:14 cinder-backup
drwxr-xr-x. 2 root root 6 4月  27 19:14 cinder-volume
[root@procontroller01 glance]#

3.3.4.4.3 cinder 相关密钥文件拷贝到 kolla-ansbile 部署节点

[cephadmin@proceph01 ceph]$ scp /etc/ceph/ceph.client.cinder.keyring root@10.3.140.11:/etc/kolla/config/cinder/cinder-backup/
root@10.3.140.11's password:
ceph.client.cinder.keyring                                                                     100%   64   121.4KB/s   00:00
[cephadmin@proceph01 ceph]$ scp /etc/ceph/ceph.client.cinder.keyring root@10.3.140.11:/etc/kolla/config/cinder/cinder-volume/
root@10.3.140.11's password:
ceph.client.cinder.keyring                                                                     100%   64   124.8KB/s   00:00
[cephadmin@proceph01 ceph]$ scp /etc/ceph/ceph.client.cinder-backup.keyring root@10.3.140.11:/etc/kolla/config/cinder/cinder-backup/
root@10.3.140.11's password:
ceph.client.cinder-backup.keyring                                                              100%   71   141.6KB/s   00:00
[cephadmin@proceph01 ceph]$

3.3.4.4.4 cinder 配置目录下添加ceph

因为ceph.conf 配置文件内容 同上 3.3.4.3.4 ,故直接复制过来即可

[root@procontroller01 glance]# cp /etc/kolla/config/glance/ceph.conf  /etc/kolla/config/cinder/
[root@procontroller01 glance]# ll /etc/kolla/config/cinder/
总用量 4
-rw-r--r--. 1 root root 100 4月  27 19:19 ceph.conf
drwxr-xr-x. 2 root root  81 4月  27 19:17 cinder-backup
drwxr-xr-x. 2 root root  40 4月  27 19:17 cinder-volume
[root@procontroller01 glance]#

3.3.4.5 配置nova 使用ceph

3.3.4.5.1 kolla ansible 部署节点新建/etc/kolla/config/nova目录

[root@procontroller01 glance]#  mkdir -p /etc/kolla/config/nova
[root@procontroller01 glance]# ll /etc/kolla/config/
总用量 0
drwxr-xr-x. 4 root root 65 4月  27 19:19 cinder
drwxr-xr-x. 2 root root 57 4月  27 19:14 glance
drwxr-xr-x. 2 root root  6 4月  27 19:20 nova
[root@procontroller01 glance]#

3.3.4.5.2 拷贝ceph.client.cinder.keyring 到 /etc/kolla/config/nova

[cephadmin@proceph01 ceph]$ scp /etc/ceph/ceph.client.cinder.keyring root@10.3.140.11:/etc/kolla/config/nova/
root@10.3.140.11's password:
ceph.client.cinder.keyring                                                                     100%   64   126.0KB/s   00:00
[cephadmin@proceph01 ceph]$

3.3.4.5.3 配置ceph.conf

[root@procontroller01 glance]# cp /etc/kolla/config/glance/ceph.conf  /etc/kolla/config/nova/
[root@procontroller01 glance]# ll /etc/kolla/config/nova/
总用量 8
-rw-r--r--. 1 root root  64 4月  27 19:21 ceph.client.cinder.keyring
-rw-r--r--. 1 root root 100 4月  27 19:23 ceph.conf
[root@procontroller01 glance]#

3.3.4.5.4 配置nova.conf

umap可以启用trim
writeback设置虚拟机的disk cache

[root@procontroller01 glance]# vim /etc/kolla/config/nova/nova.conf
[root@procontroller01 glance]#

内容如下：
[root@procontroller01 glance]# cat /etc/kolla/config/nova/nova.conf
[libvirt]
hw_disk_discard = unmap
disk_cachemodes="network=writeback"
cpu_mode=host-passthrough
[root@procontroller01 glance]#

3.3.5 kolla-ansible 安装openstack train

3.3.5.1 预检查