Kubernetes作为目前最炙手可热的容器编排软件，受到越来越多互联网公司的喜爱，之所以会出现这样的现象，主要还是因为这套编排软件不仅功能强大，还非常方便进行二次开发，整个Kubernetes的生态圈非常活跃。有了Kubernetes这套“骨架”，可以在上面运行非常多的中间件和应用程序。Apiserver，Controller Manager，Scheduler，Proxy可以认为是这套“骨架”的组成部分，而Etcd便是这套系统的底座，整个系统的数据都存储在它当中，保证了Etcd的稳定性整个容器编排系统的稳定性就有了保证。
在正式开始之前，说明下实验环境，五台主机组成Kubernetes集群。
- 控制平面主机：p1.xufu.xyz、p2.xufu.xyz、p3.xufu.xyz
- 工作主机：p4.xufu.xyz、p5.xufu.xyz
- 主机系统：CentOS Linux release 7.9.2009 (Core)
- Etcd版本：3.5.1
Etcd
- 简介
它是一种键值数据库，Kubernetes将自身的状态数据都存储在Etcd中，这些数据包含deployment，pod，daemonset等各种对象。整个集群组件中只有Apiserver会直接和Etcd通信，其他组件都是直接和Apiserver通信。在生产环境中，最好以集群的方式运行Etcd，官方建议运行五个节点组成集群，这样可以在丢失两个节点的情况下， 保证数据的可用性。
- 访问
Etcd中存放有不少敏感数据，所以对它的访问需要使用公钥和私钥。为了方便访问，在家目录文件“.bashrc”配置如下命令别名。
<pre class="prettyprint hljs nginx" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">alias ectl='etcdctl --endpoints p1.xufu.xyz:2379 \ --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \ --cacert=/etc/kubernetes/pki/etcd/ca.crt'</pre>显示当前Etcd存储的所有Key。
<pre class="prettyprint hljs vim" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">ectl get / --prefix --keys-only</pre>- 备份
使用kubeadm部署的集群，Etcd默认将数据存放在宿主机的目录/var/lib/etcd下。
Etcd数据目录
Etcd的版本2和3差别很大，因笔者使用的是3版本，备份命令是这样（在任一控制平面主机操作均可），命令执行不用对Etcd做关闭操作。
<pre class="prettyprint hljs nginx" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">ectl snapshot save snapshot-$(date +%F)</pre>查看备份的快照。
快照信息
- 恢复
恢复操作相比较备份操作有稍许麻烦。下面步骤需要在三台控制平面主机上都操作。
1，关闭kubelet，否则它会自动重启apiserver。
<pre class="prettyprint hljs nginx" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">systemctl stop kubelet.service</pre>2，关闭apiserver容器，使用docker命令找到容器，直接stop。
<pre class="prettyprint hljs nginx" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">docker stop apiservername</pre>3，重命名目录/var/lib/etcd，创建新的，并修改目录权限。
<pre class="prettyprint hljs awk" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">mv /var/lib/etcd /var/lib/etcd_oldmkdir /var/lib/etcdchmod 700 /var/lib/etcd</pre>4，使用备份的快照，恢复数据。
<pre class="prettyprint hljs awk" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">ectl snapshot restore /data/etcd_backup/etcd-snapshot-2022-08-03.db \ --name p1.xufu.xyz \ --initial-cluster "etcd-0=https://p1.xufu.xyz:2380,etcd-1=https://p2.xufu.xyz:2380,etcd-2=https://p3.xufu.xyz:2380" \ --initial-advertise-peer-urls https://p1.xufu.xyz:2380 \ --data-dir=/var/lib/etcd</pre>p2和p3节点类似。
5，启动三台etcd，确认下集群是否健康。
<pre class="prettyprint hljs vim" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">[root@p1 etcd]# ectl --endpoints=https://p1.xufu.xyz:2379,https://p2.xufu.xyz:2379,https://p3.xufu.xyz:2379 endpoint healthhttps://p2.xufu.xyz:2379 is healthy: successfully committed proposal: took = 100.193247msp1.xufu.xyz:2379 is healthy: successfully committed proposal: took = 111.234607mshttps://p1.xufu.xyz:2379 is healthy: successfully committed proposal: took = 113.788721mshttps://p3.xufu.xyz:2379 is healthy: successfully committed proposal: took = 118.403445ms</pre>确认没问题后，启动apiserver容器。
至此，整个恢复备份过程操作完毕。
