Kubernetes 系列（4）使用 kubeadm 部署高可用集群

目录

文章目录

• ​​目录​​
• ​​高可用集群部署拓扑​​
• ​​1、网络代理配置​​
• ​​2、Load Balancer 环境准备​​
• ​​3、Kubernetes Cluster 环境准备​​

• ​​安装 Container Runtime​​
• ​​安装 kubeadm、kubelet 和 kubectl​​

• ​​4、初始化 Master 主控制平面节点​​

• ​​kubeadm init 的工作流​​
• ​​执行初始化​​

• ​​（可选）清理或重新进行初始化​​
• ​​5、添加 Master 冗余控制平面节点​​
• ​​6、添加 Node 工作负载节点​​
• ​​7、安装 CNI 网络插件​​
• ​​8、安装 Metrics Server​​

• ​​Kubernetes Metrics Server​​
• ​​启用 API Aggregator​​
• ​​安装 Metrics Server​​

• ​​9、安装 Dashboard GUI​​
• ​​10、访问 Dashboard UI​​
• ​​11、通过 NFS 实现持久化存储​​

• ​​NFS Server 安装​​
• ​​NFS Client 挂载​​
• ​​Kubernetes 部署 nfs-client-provisioner​​

高可用集群部署拓扑

官方文档：https://kubernetes.io/zh/docs/setup/production-environment/

• 基础设施：OpenStack
• 虚拟机集群：3 Master、2 Node、2 Load Balancer
• 计算资源：x86-64 processor、2CPU、2GB RAM、20GB free disk space
• 操作系统：CentOS 7.x+
• 版本：Kubernetes 1.18.14
• Container Runtime：Docker

1、网络代理配置

因为要科学上网，所以需要对 HTTP/S Proxy 和 No Proxy 进行精心的配置，否则要么下不下来软件，要么出现网络连通性的错误。

export https_proxy=http://{proxy_ip}:7890 http_proxy=http://{proxy_ip}:7890 all_proxy=socks5://{proxy_ip}:7890 no_proxy=localhost,127.0.0.1,{apiserver_endpoint_ip},{k8s_mgmt_network_ip_pool},{pod_network_ip_pool},{service_network_ip_pool}

2、Load Balancer 环境准备

基于 OpenStack Octavia LBaaS 来提供 HA Load Balancer，也可以手动的配置 keepalived and haproxy（https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#options-for-software-load-balancing）。

• VIP 选择 kube-mgmt-subnet

  

• Listener 选择 TCP :6443 Socket（kube-apiserver 的监听端口）

  

• Members 选择 3 个 k8s-master

  

• Monitor 同样选择 TCP :6443 Socket

  

注意：创建好 Load Balancer 之后，首先要测试一下 TCP 反向代理运行正常。由于 apiserver 现在尚未运行，所以预期会出现一个连接拒绝错误。在我们初始化了第一个控制平面节点之后，要记得再次进行测试。

# nc -v LOAD_BALANCER_IP PORT
nc -v 192.168.0.100 6443

3、Kubernetes Cluster 环境准备

注意：在所有节点上执行以下操作。

• 科学上网。
• 添加全节点的 Hostname 解析。

# vi /etc/hosts

192.168.0.100 kube-apiserver-endpoint
192.168.0.148 k8s-master-1
192.168.0.112 k8s-master-2
192.168.0.193 k8s-master-3
192.168.0.208 k8s-node-1
192.168.0.174 k8s-node-2

• 开启全节点之间的 SSH 免密登录。
• 禁用 Swap 交换分区，为了保证 kubelet 正常工作。
• 确保 iptables 工具不使用 nftables 后端，nftables 后端与当前的 kubeadm 软件包不兼容，它会导致重复的防火墙规则并破坏 kube-proxy。
• 确保节点之间的网络联通性。
  
• 关闭 SELinux，为了允许容器访问主机的文件系统。

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

• 在 RHEL/CentOS 7 上为了保证 kube-proxy 控制的数据流量必须进过 iptables 的处理来进行本地路由，所以要确保 sysctl 配置中的 net.bridge.bridge-nf-call-iptables 被设置为 1。

# 确保加载了 br_netfilter 模块。
modprobe br_netfilter
lsmod | grep br_netfilter

# 确保 sysctl 配置，将 Bridge 的 IPv4 流量传递到 iptables 的 Chain（链）
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

• 安装基础依赖软件：

yum install ebtables ethtool ipvsadm -y

安装 Container Runtime

注意：当 Linux 使用 systemd 时，会创建一个 cgroup，此时需要保证 Container Runtime、kubelet 和 systemd 使用的是同一个 cgroup，否则会出现不可预测的问题。为此，我们需要将 Container Runtime、kubelet 配置成使用 systemd 来作为 cgroup 驱动，以此使系统更为稳定。

对于 Docker 而言，设置 ​​native.cgroupdriver=systemd​​ 选项即可。

• 安装：

# 安装依赖包
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

# 新增 Docker 仓库
sudo yum-config-manager --add-repo \
  https://download.docker.com/linux/centos/docker-ce.repo

# 安装 Docker CE
sudo yum update -y && sudo yum install -y \
  containerd.io-1.2.13 \
  docker-ce-19.03.11 \
  docker-ce-cli-19.03.11

• 配置：

# 创建 /etc/docker 目录
sudo mkdir /etc/docker

# 设置 Docker daemon
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

• 重启：

# Create /etc/systemd/system/docker.service.d
sudo mkdir -p /etc/systemd/system/docker.service.d

# 重启 Docker
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker

sudo systemctl status docker

安装 kubeadm、kubelet 和 kubectl

注意：kubeadm 是 Kubernetes Cluster 的部署工具，但 kubeadm 不能用于安装、管理 kubelet 或 kubectl，所以我们需要收到安装它们，并且确保三者的版本仓库是一致的。

• 更新 Kubernetes YUM 仓库：

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

• 安装：

# 查询版本
$ yum list kubelet kubeadm kubectl --showduplicates | grep 1.18.14 | sort -r
kubelet.x86_64                       1.18.14-0                       kubernetes
kubectl.x86_64                       1.18.14-0                       kubernetes
kubeadm.x86_64                       1.18.14-0                       kubernetes

# 安装指定版本
yum install -y kubelet-1.18.14 kubeadm-1.18.14 kubectl-1.18.14 --disableexcludes=kubernetes

# 确定版本一致
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.14", GitCommit:"89182bdd065fbcaffefec691908a739d161efc03", GitTreeState:"clean", BuildDate:"2020-12-18T12:08:45Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.14", GitCommit:"89182bdd065fbcaffefec691908a739d161efc03", GitTreeState:"clean", BuildDate:"2020-12-18T12:11:25Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

$ kubelet --version
Kubernetes v1.18.14

• 配置：上面我们提到过，需要将 Container Runtime、kubelet 配置成使用 systemd 来作为 cgroup 驱动，以此使系统更为稳定。

# vi /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd

• 启动：

$ systemctl daemon-reload
$ systemctl restart kubelet
$ systemctl enable --now kubelet
$ systemctl status kubelet

注意：kubelet.sercice 每隔几秒就会重启一次，循环等待 kubeadm 的指令。

4、初始化 Master 主控制平面节点

kubeadm init 的工作流

kubeadm init 命令通过执行下列步骤来启动一个 Kubernetes Master：

1. 预检测系统状态：当出现 ERROR 时就退出 kubeadm，除非问题得到解决或者显式指定了 ​​--ignore-preflight-errors=<错误列表>​​ 参数。此外，也会出现 WARNING。
2. 生成一个自签名的 CA 证书来为每个系统组件建立身份标识：可以显式指定 ​​--cert-dir​​ CA 中心目录（默认为 /etc/kubernetes/pki），在该目录下方式 CA 证书、密钥等文件。API Server 证书将为任何 ​​--apiserver-cert-extra-sans​​ 参数值提供附加的 SAN 条目，必要时将其小写。
3. 将 kubeconfig 文件写入 /etc/kubernetes/ 目录：以便 kubelet、Controller Manager 和 Scheduler 用来连接到 API Server，它们都有自己的身份标识，同时生成一个名为 admin.conf 的独立的 kubeconfig 文件，用于管理操作。
4. 为 API Server、Controller Manager 和 Scheduler 生成 static Pod 的清单文件：存放在 /etc/kubernetes/manifests 下，kubelet 会轮训监视这个目录，在启动 Kubernetes 时用于创建系统组件的 Pod。假使没有提供一个外部的 etcd 服务的话，也会为 etcd 生成一份额外的 static Pod 清单文件。

待 Master 的 static Pods 都运行正常后，kubeadm init 的工作流程才会继续往下执行。

5. 对 Master 使用 Labels 和 Stain mark（污点标记）：以此隔离生产工作负载不会调度到 Master 上。
6. 生成 Token：将来其他的 Node 可使用该 Token 向 Master 注册自己。也可以显式指定 ​​--token​​ 提供 Token String。
7. 为了使 Node 能够遵照​​启动引导令牌（Bootstrap Tokens）​​和 ​​TLS 启动引导（TLS bootstrapping）​​这两份文档中描述的机制加入到 Cluster 中，kubeadm 会执行所有的必要配置：

1. 创建一个 ConfigMap 提供添加 Node 到 Cluster 中所需的信息，并为该 ConfigMap 设置相关的 RBAC 访问规则。
2. 允许启动引导令牌访问 CSR 签名 API。
3. 配置自动签发新的 CSR 请求。

8. 通过 API Server 安装一个 DNS 服务器（CoreDNS）和 kube-proxy：注意，尽管现在已经部署了 DNS 服务器，但直到安装 CNI 时才调度它。

执行初始化

注意 1：因为我们要部署高可用集群，所以必须使用选项 ​​--control-plane-endpoint​​ 指定 API Server 的 HA Endpoint。

注意 2：由于 kubeadm 默认从 k8s.grc.io 下载所需镜像，因此可以通过 ​​--image-repository​​ 指定阿里云的镜像仓库。

注意 3：如果显式指定 ​​--upload-certs​​，则意味着在扩展冗余 Master 时，你必须要手动地将 CA 证书从主控制平面节点复制到将要加入的冗余控制平面节点上，推荐使用。

• 初始化：

kubeadm init \
  --control-plane-endpoint "192.168.0.100" \
  --kubernetes-version "1.18.14" \
  --pod-network-cidr "10.0.0.0/8" \
  --service-cidr "172.16.0.0/16" \
  --token "abcdef.0123456789abcdef" \
  --token-ttl "0" \
  --image-repository registry.aliyuncs.com/google_containers \
  --upload-certs

W1221 00:02:43.240309   10942 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.14
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.16.0.1 192.168.0.148 192.168.0.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.0.148 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.0.148 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1221 00:02:47.773223   10942 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1221 00:02:47.774303   10942 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.117265 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
    --control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1

• 查看 Pods：检查 Master 的组件是否齐全。

# 配置 kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

$ kubectl get pod -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
coredns-7ff77c879f-fh9vb               0/1     Pending   0          23m
coredns-7ff77c879f-qmk7z               0/1     Pending   0          23m
etcd-k8s-master-1                      1/1     Running   0          24m
kube-apiserver-k8s-master-1            1/1     Running   0          24m
kube-controller-manager-k8s-master-1   1/1     Running   0          24m
kube-proxy-7hx55                       1/1     Running   0          23m
kube-scheduler-k8s-master-1            1/1     Running   0          24m

• 查看 Images：

$ docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/kube-proxy                v1.18.14            8e6bca1d4e68        2 days ago          117MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.18.14            f17e261f4c8a        2 days ago          173MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.18.14            b734a959c6fb        2 days ago          162MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.18.14            95660d582e82        2 days ago          95.3MB
registry.aliyuncs.com/google_containers/pause                     3.2                 80d28bedfe5d        10 months ago       683kB
registry.aliyuncs.com/google_containers/coredns                   1.6.7               67da37a9a360        10 months ago       43.8MB
registry.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        14 months ago       288MB

• 查看 Containers：

$ docker ps -a
CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS              PORTS               NAMES
f9a068b890d7        8e6bca1d4e68                                        "/usr/local/bin/kube…"   2 minutes ago       Up 2 minutes                            k8s_kube-proxy_kube-proxy-7hx55_kube-system_aacb0da3-16ec-414c-b138-856e2b470bb9_0
3b6adfa0b1a5        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-proxy-7hx55_kube-system_aacb0da3-16ec-414c-b138-856e2b470bb9_0
dcc47de63e50        f17e261f4c8a                                        "kube-apiserver --ad…"   3 minutes ago       Up 3 minutes                            k8s_kube-apiserver_kube-apiserver-k8s-master-1_kube-system_c693bd1fadf036d8e2e4df0afd49f062_0
53afb7fbe8c0        b734a959c6fb                                        "kube-controller-man…"   3 minutes ago       Up 3 minutes                            k8s_kube-controller-manager_kube-controller-manager-k8s-master-1_kube-system_f75424d466cd7197fb8095b0f59ea8d9_0
a4101a231c1b        303ce5db0e90                                        "etcd --advertise-cl…"   3 minutes ago       Up 3 minutes                            k8s_etcd_etcd-k8s-master-1_kube-system_f85e02734d6479f3bb3e468eea87fd3a_0
197f510ff6c5        95660d582e82                                        "kube-scheduler --au…"   3 minutes ago       Up 3 minutes                            k8s_kube-scheduler_kube-scheduler-k8s-master-1_kube-system_0213a889f9350758ac9847629f75db19_0
3a4590590093        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-controller-manager-k8s-master-1_kube-system_f75424d466cd7197fb8095b0f59ea8d9_0
4bbdc99a7a68        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-apiserver-k8s-master-1_kube-system_c693bd1fadf036d8e2e4df0afd49f062_0
19488127c269        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_etcd-k8s-master-1_kube-system_f85e02734d6479f3bb3e468eea87fd3a_0
e67d2f7a27b0        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-scheduler-k8s-master-1_kube-system_0213a889f9350758ac9847629f75db19_0

• 测试 API Server LB 是否正常：

$ nc -v 192.168.0.100 6443
Connection to 192.168.0.100 port 6443 [tcp/sun-sr-https] succeeded!

注意：上述 Token 的过期时间是 24 小时，如果希望在 24 小时之后继续添加不通的节点，则需要重新生产 Token：

# 新建 Token
kubeadm token create
# output: 5didvk.d09sbcov8ph2amjw

# 新建 --discovery-token-ca-cert-hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'
# output: 8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78

（可选）清理或重新进行初始化

要再次运行 kubeadm init，你必须首先卸载集群，可以在 Master 上触发尽力而为的清理：

kubeadm reset

Reset 过程不会重置或清除 iptables 规则或 IPVS 表。如果你希望重置 iptables 或 IPVS，则必须手动进行：

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ipvsadm -C

根据需求调整参数，重新进行初始化：

kubeadm init <args>

或许，彻底删除节点：

kubectl delete node <node name>

5、添加 Master 冗余控制平面节点

在第一个 Master 初始化完毕之后，我们就可以继续添加冗余 Master 节点了。

• 添加 k8s-master-2：

kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
    --control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-2 localhost] and IPs [192.168.0.112 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-2 localhost] and IPs [192.168.0.112 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.16.0.1 192.168.0.112 192.168.0.100]
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
W1221 00:30:18.978564   27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1221 00:30:18.986650   27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1221 00:30:18.987613   27668 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
{"level":"warn","ts":"2020-12-21T00:30:34.018+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"passthrough:///https://192.168.0.112:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node k8s-master-2 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master-2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

    Run 'kubectl get nodes' to see this node join the cluster.

• 添加 k8s-master-3：

kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1 \
    --control-plane --certificate-key 463868e92236803eb8fdeaa3d7b0ada67cf0f882c45974682c6ac2f20be1d544

• 检查 Master 节点数：

# 配置 kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

$ kubectl get nodes
NAME           STATUS     ROLES    AGE     VERSION
k8s-master-1   NotReady   master   35m     v1.18.14
k8s-master-2   NotReady   master   8m14s   v1.18.14
k8s-master-3   NotReady   master   2m30s   v1.18.14

6、添加 Node 工作负载节点

部署完高可用的 Master 控制平面之后，我们就可以注册任意个 Node 工作负载节点了。

• 添加 Node：

kubeadm join 192.168.0.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:88dc9773b5dfc0cde6082314a1a4a9bbdb6ddfd3f1f84a7113581a3b07e839e1

W1221 00:39:36.256784   29495 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

• 检查 Node：

$ kubectl get nodes
NAME           STATUS     ROLES    AGE     VERSION
k8s-master-1   NotReady   master   37m     v1.18.14
k8s-master-2   NotReady   master   10m     v1.18.14
k8s-master-3   NotReady   master   4m24s   v1.18.14
k8s-node-1     NotReady   <none>   51s     v1.18.14
k8s-node-2     NotReady   <none>   48s     v1.18.14

7、安装 CNI 网络插件

我们选择使用 Calico SDN 方案。官方文档：https://docs.projectcalico.org/about/about-calico

注意：

1. Pod 网络不得与任何主机网络重叠：所以我们在执行 kubeadm init 时显式指定了​​--pod-network-cidr ​​ 参数。
2. 确保 CNI 网络插件支持 RBAC（基于角色的访问控制）。
3. 确保 CNI 支持 IPv6 或 IPv4v6，当你需要使用的时候。
4. 在 OpenStack 环境中要关闭 “端口安全” 功能，否则 Calico IPIP 隧道无法打通。
   

• 安装：

$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created

• 检查 Calico Pods：

$ watch kubectl get pod --all-namespaces

Every 2.0s: kubectl get pod --all-namespaces                                                                                                                                        Mon Dec 21 13:12:30 2020

NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7dbc97f587-nqrxv   1/1     Running   0          9m34s
kube-system   calico-node-47xmr                          1/1     Running   0          9m34s
kube-system   calico-node-8zwbg                          1/1     Running   0          9m34s
kube-system   calico-node-dj4qt                          1/1     Running   0          9m34s
kube-system   calico-node-glqqj                          1/1     Running   0          9m34s
kube-system   calico-node-jb4t4                          1/1     Running   0          9m34s
kube-system   coredns-7ff77c879f-fh9vb                   1/1     Running   0          13h
kube-system   coredns-7ff77c879f-qmk7z                   1/1     Running   0          13h
kube-system   etcd-k8s-master-1                          1/1     Running   0          13h
kube-system   etcd-k8s-master-2                          1/1     Running   0          12h
kube-system   etcd-k8s-master-3                          1/1     Running   0          12h
kube-system   kube-apiserver-k8s-master-1                1/1     Running   0          13h
kube-system   kube-apiserver-k8s-master-2                1/1     Running   0          12h
kube-system   kube-apiserver-k8s-master-3                1/1     Running   0          12h
kube-system   kube-controller-manager-k8s-master-1   1/1     Running   1          13h
kube-system   kube-controller-manager-k8s-master-2   1/1     Running   0          12h
kube-system   kube-controller-manager-k8s-master-3   1/1     Running   0          12h
kube-system   kube-proxy-7hx55                           1/1     Running   0          13h
kube-system   kube-proxy-8dmc4                           1/1     Running   0          12h
kube-system   kube-proxy-9clqs                           1/1     Running   0          12h
kube-system   kube-proxy-cq5tq                           1/1     Running   0          12h
kube-system   kube-proxy-pm79q                           1/1     Running   0          12h
kube-system   kube-scheduler-k8s-master-1                1/1     Running   1          13h
kube-system   kube-scheduler-k8s-master-2                1/1     Running   0          12h
kube-system   kube-scheduler-k8s-master-3                1/1     Running   0          12h

• 检查 Cluster 节点的状态：安装了 CNI 之后节点的状态应该是 Ready 的。

$ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
k8s-master-1   Ready    master   13h   v1.18.14
k8s-master-2   Ready    master   12h   v1.18.14
k8s-master-3   Ready    master   12h   v1.18.14
k8s-node-1     Ready    <none>   12h   v1.18.14
k8s-node-2     Ready    <none>   12h   v1.18.14

8、安装 Metrics Server

Kubernetes Metrics Server

Kubernetes Metrics Server 是 Cluster 的核心监控数据的聚合器，kubeadm 默认是不部署的。

Metrics Server 供 Dashboard 等其他组件使用，是一个扩展的 APIServer，依赖于 API Aggregator。所以，在安装 Metrics Server 之前需要先在 kube-apiserver 中开启 API Aggregator。

• Metrics API 只可以查询当前的度量数据，并不保存历史数据。
• Metrics API URI 为 /apis/metrics.k8s.io/，在 k8s.io/metrics 下维护。
• 必须部署 metrics-server 才能使用该 API，metrics-server 通过调用 kubelet Summary API 获取数据。

使用 Metrics Server 有必备两个条件：

1. API Server 启用 Aggregator Routing 支持。否则 API Server 不识别请求：

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

2. API Server 能访问 Metrics Server Pod IP。否则 API Server 无法访问 Metrics Server：

E1223 07:23:04.330206       1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: Get https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

启用 API Aggregator

API Aggregation 允许在不修改 Kubernetes 核心代码的同时扩展 Kubernetes API，即：将第三方服务注册到 Kubernetes API 中，这样就可以通过 Kubernetes API 来访问第三方服务了，例如：Metrics Server API。

注：另外一种扩展 Kubernetes API 的方法是使用 CRD（Custom Resource Definition，自定义资源定义）。

• 检查 API Server 是否开启了 Aggregator Routing：查看 API Server 是否具有​​--enable-aggregator-routing=true​​ 选项。

$ ps -ef | grep apiserver
root     23896 29500  0 12:40 pts/0    00:00:00 grep --color=auto apiserver
root     28613 28551  1 12月21 ?      01:05:29 kube-apiserver --advertise-address=192.168.0.112 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=172.16.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

• 修改每个 API Server 的 kube-apiserver.yaml 配置开启 Aggregator Routing：修改 manifests 配置后会 API Server 会自动重启生效。

$ vi /etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
  containers:
  - command:
...
  - --enable-aggregator-routing=true

安装 Metrics Server

• 检查 Cluster 是否安装了 Metrics Server：

$ kubectl top pods
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

• 部署 Metrics Server：

# 下载 YAML 文件
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml

# 编辑修改 metrics-server 的启动参数：
#  --kubelet-insecure-tls 跳过 TLS 认证，否则会出现 x509 的认证问题，用于测试环境。
#  --kubelet-preferred-address-types=InternalIP 使用 Node IP 进行通信。
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP
        - --kubelet-use-node-status-port
        - --kubelet-insecure-tls

# 部署
$ kubectl apply -f components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

注意：如果出现了 ErrImagePull 的问题，那么意味着 k8s.gcr.io/metrics-server/metrics-server:v0.4.1 镜像下载失败了：

$ docker pull k8s.gcr.io/metrics-server/metrics-server:v0.4.1
Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

这时候就需要我们在每个节点上都手动的下载镜像了：

$ docker pull bitnami/metrics-server:0.4.1

$ docker tag bitnami/metrics-server:0.4.1 k8s.gcr.io/metrics-server/metrics-server:v0.4.1

$ docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
bitnami/metrics-server                                            0.4.1               4fb6df85a88d        6 hours ago         171MB
k8s.gcr.io/metrics-server/metrics-server                          v0.4.1              4fb6df85a88d        6 hours ago         171MB

然后再次执行 Metrics Server 的部署指令。

• 检查 Metrics Server Service：

$ kubectl get svc --all-namespaces | grep metrics-server
kube-system   metrics-server   ClusterIP   172.16.128.176   <none>        443/TCP                  5h55m

• 检查 API Server 是否可以连通 Metrics Server：

$ kubectl describe svc metrics-server -n kube-system
Name:              metrics-server
Namespace:         kube-system
Labels:            k8s-app=metrics-server
Annotations:       Selector:  k8s-app=metrics-server
Type:              ClusterIP
IP:                172.16.128.176
Port:              https  443/TCP
TargetPort:        https/TCP
Endpoints:         10.171.248.214:4443
Session Affinity:  None
Events:            <none>

# 在 Master Node 上 Ping。
$ ping 10.171.248.214
64 bytes from 10.171.248.214: icmp_seq=1 ttl=63 time=0.282 ms

• 检查 Metrics Server：

$ kubectl top nodes
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master-1   174m         8%     1156Mi          66%
k8s-master-2   123m         6%     1134Mi          65%
k8s-master-3   104m         5%     1075Mi          61%
k8s-node-1     78m          3%     853Mi           49%
k8s-node-2     78m          3%     824Mi           47%

9、安装 Dashboard GUI

默认情况下不会部署 Dashboard，需要我们手动安装。在安装 Dashboard 之前我们还需要确保安装了 Metrics Server。否则，错误提示我们在 Cluster 中没有安装 Metrics Server。

$ kubectl logs -f -n kubernetes-dashboard dashboard-metrics-scraper-6b4884c9d5-qppkd
192.168.0.208 - - [21/Dec/2020:07:54:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.18"
{"level":"error","msg":"Error scraping node metrics: the server is currently unable to handle the request (get nodes.metrics.k8s.io)","time":"2020-12-21T07:54:09Z"}

$ kubectl logs -f -n kubernetes-dashboard kubernetes-dashboard-7b544877d5-p6g8t
2020/12/21 07:54:48 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.

另外，官方 YAML 部署的 Dashboard 的 Service 也不是 NodePort 类型的，我们需要手动修改。

• 编辑

$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

$ vi recommended.yaml

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30000
  selector:
    k8s-app: kubernetes-dashboard

---

• 部署：出于安全访问的考虑，默认情况下，Dashboard 会使用最基础的 RBAC 配置进行部署。

$ kubectl create -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

• 检查 Dashboard Pods：

$ kubectl get pods --all-namespaces | grep dashboard
kubernetes-dashboard   dashboard-metrics-scraper-6b4884c9d5-ctn6h   1/1     Running   0          79s
kubernetes-dashboard   kubernetes-dashboard-7b544877d5-sghsf        1/1     Running   0          79s

• 检查 Dashboard Services：检查是否与 YAML 描述的一致。

$ kubectl get svc -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   172.16.165.243   <none>        8000/TCP        2m32s
kubernetes-dashboard        NodePort    172.16.219.86    <none>        443:30000/TCP   2m33s

10、访问 Dashboard UI

因为 kubernetes-dashboard Service 是 NodePod 类型，所以我们使用任意节点 IP + NodePortNum 就可以访问了。

注意：NodePort 并非 VIP HA LB 的方式，所以我们可以使用高可用部署环境中的 Load Balancer 进行代理。

由于 Dashboard 默认是自建的 HTTPS 证书，该证书是不受浏览器信任的，强制跳转即可。

登录 Dashboard 的时候支持 Kubeconfig 和 Token 两种认证方式，由于 Kubeconfig 中也依赖 Token 字段，所以生成 Token 这个步骤是必不可少的。

• 创建 Dashboard Admin 管理员：

$ cat dashboard-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: dashboard-admin
  namespace: kubernetes-dashboard

$ kubectl create -f ./dashboard-admin.yaml
serviceaccount/dashboard-admin created

# CLI: kubectl create serviceaccount dashboard-admin -n kubernetes-dashboard

• 为 Dashboard Admin 分配权限：

$ cat dashboard-admin-bind-cluster-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin-bind-cluster-role
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: dashboard-admin
  namespace: kubernetes-dashboard

$ kubectl create -f ./dashboard-admin-bind-cluster-role.yaml
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin-bind-cluster-role created

# CLI: kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin

• 生成 Token：

$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep dashboard-admin | awk '{print $1}')
Name:         dashboard-admin-token-bc2dh
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: 3c26e908-49b2-4de8-8f08-699dba736a56

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IkV3a2JIWGhCdTFTSEFpd2hxam1WM29RTzQwcXdlT2dLYlRTRFU2TGotNHMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tYmMyZGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2MyNmU5MDgtNDliMi00ZGU4LThmMDgtNjk5ZGJhNzM2YTU2Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.v24ad9kFp4_kEUo4pPVRxqHxe6rXCv4kZryR3QoWrCKTdHM2FmJPrU1q_w5wGyXVShQQGyBlQJpdAAX5RAIy4gVvLLbc7W9qE74jHVijb4pBp9j8r-lsYpGvpiXadVCqXYCXHBxQSeYKXNM6hwDGKFGxQZ2LDA8K_j590fVeozSxM1RBJxetM4KfF0KomDurOhjITu0rortufeKhkOvRcoB0EikPkwqbU0Q5Ip7m_YIYHZsEqo292GieTyfZIM0KRLGGn4GX53qTbgk1bhQYsequUjs3sKw5Ng8vSOq8NPX0vB88Mjl0fzPWtKvnvpWokg1JL_fW4-qKA7HbUcenXQ

• 访问：
  

11、通过 NFS 实现持久化存储

NFS Server 安装

$ yum install -y nfs-utils
$ yum install -y rpcbind

$ mkdir /public
$ vi /etc/exports
  /public 192.168.186.*

$ systemctl start rpcbind && systemctl enable rpcbind && systemctl status rpcbind
$ systemctl start nfs-server && systemctl enable nfs-server && systemctl status nfs-server

NFS Client 挂载

$ mkdir /mnt/public

$ vi /etc/fstab
  192.168.186.198:/public /mnt/public nfs defaults 0 0

$ mount -a

Kubernetes 部署 nfs-client-provisioner

• 下载 nfs-client

git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git

• 创建 ServiceAccount

$ NS=$(kubectl config get-contexts|grep -e "^\*" |awk '{print $5}')
$ NAMESPACE=${NS:-default}
$ cd nfs-subdir-external-provisioner/deploy
$ sed -i'' "s/namespace:.*/namespace: $NAMESPACE/g" ./deploy/rbac.yaml ./deployment.yaml
$ kubectl create -f ./rbac.yaml

• 部署 NFS Provisioner

$ vi deployment.yaml
...
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.186.198  # NFS Server IP
            - name: NFS_PATH
              value: /public          # Share Dir Path
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.186.198   # NFS Server IP
            path: /public             # Share Dir Path
            
 
# 创建 NFS Provisioner
$ kubectl apply -f deployment.yaml

• 创建 NFS Storage Class

kubectl apply -f class.yaml