k8s知识梳理--进阶

网络#

Kubernetes - 网络模型原则

在不使用网络地址转换 (NAT) 的情况下，集群中的 Pod 能够与任意其他 Pod 进行通信

在不使用网络地址转换 (NAT) 的情况下，在集群节点上运行的程序能与同一节点上的任何 Pod 进行通信

每个 Pod 都有自己的 IP 地址 (IP-per-Pod)，并且任意其他 Pod 都可以通过相同的这个地址访问它

借助 CNI 标准，Kubernetes 可以实现容器网络问题的解决。通过插件化的方式来集成各种网络插件，实现集群内部网终相互通信，只要实现 CNI 标准中定义的核心接口操作 (ADD, 将容器添加到网络；DEL，从网络中删除一个容器；CHECK, 检查容器的网络是否符合预期等)。CNI 插件通常聚焦在容器到容器的网络通信。

CNl 的接口并不是指 HTTP,gRPC 这种接口，CNl 接口是指对可执行程序的调用 (exec) 可执行程序，Kubernetes 节点默认的 CN 插件路径为 /opt/cni/bin

CNI 通过 JSON 格式的配置文件来描述网络配置，当需要设置容器网络时，由容器运行时负责执行 CN! 插件，并通过 CNl 插件的标准输入 (stdi) 来传递配置文件信息，通过标准输出 (stdout) 接收插件的执行结果。从网络插件功能可以分为五类：

Main 插件，创建具体网络设备 (bridge: 网桥设备，连接 container 和 host；ipvlan: 为容器增加 ipvlan 网卡；loopback设备；macvlan: 为容器创建一个 MAC 地址；ptp: 创建一对 VethPair；vlan: 分配一个 vlan 设备；host-device: 将已存在的设备移入容器内)
IPAM 插件：负责分配 IP 地址 (dhcp: 容器向 DHCP 服务器发起请求，给 Pod 发放或回收 P 地址；host-local: 使用预先配置的 IP 地址段来进行分配；static: 为容器分配一个静态 IPv4/IPv6 地址主要用于 debug)
META 插件：其他功能的插件 (tuning: 通过 sysctl 调整网络设备参数；portmap: 通过 iptables 配置端口映射；bandwidth: 使用 Token Bucket Filter 来限流；sbr: 为网卡设置 source based routing；firewall：通过 iptables 给容器网络的进出流量进行限制)
Windows 插件：专门用于 Windows 平台的 CNl 插件 (win-bridge 与 win-overlay 网络插件)
第三方网络插件：第三方开源的网络插件众多，每个组件都有各自的优点及适应的场景，难以形成统一的标准组件，常用有 Flannel、Calico、Cilium、OVN 网络插件

提供商	网络模型	路由分发	网络策略	网格	外部数据存储	加密	Ingress/Egress 策略
Canal	封装（VXLAN）	否	是	否	k8s API	是	是
Flannel	封装（VXLAN）	否	否	否	k8s API	是	否
Calico	封装（VXLAN, IPIP）或未封装	是	是	是	Etcd 和 k8s API	是	是
Weave	封装	是	是	是	否	是	是
Cilium	封装（VXLAN）	是	是	是	Etcd 和 k8s API	是	是

网络模型：封装或未封装。

路由分发：一种外部网关协议，用于在互联网上交换路由和可达性信息。BGP 可以帮助进行跨集群 ρod 之间的网络。此功能对于未封装的 CNI 网络插件是必须的，并且通常由 BGP 完成。如果你想构建跨网段拆分的集群，路由分发是一个很好的功能。

网络策略：Kubernetes 提供了强制执行规则的功能，这些规则决定了哪些 service 可以使用网络策略进行相互通信。这是从 Kubernetes1.7 起稳定的功能，可以与某些网络插件一起使用。

网格：允许在不同的 Kubernetes 集群间进行 service 之间的网络通信。

外部数据存储：具有此功能的 C! 网络插件需要一个外部数据存储来存储数据。

加密：允许加密和安全的网络控制和数据平面。

Ingress/Egress 策略：允许你管理 Kubernetes 和非 Kubernetes 通信的路由控制。

Calico 是一个纯三层的虚拟网络，它没有复用 docker 的 docker0 网桥，而是自己实现的，calico 网络不对数据包进行额外封装，不需要 NAT 和端口映射

Calico 架构：

Felix

管理网络接口编写路由
编写 ACL
报告状态

bird (BGP Client)

BGP Client 将通过 BGP 协议广播告诉剩余 calico 节点，从而实现网络互通

confd

通过监听 etcd 以了解 BGP 配置和全局默认值的更改。Confd 根据 ETCD 虫数据的更新，动态生成 BRD 配置文件文件更改时 confd 触发 BlRD 重新加载

Calico 网络模式 --VXLAN：

什么是 VXLAN?

VXLAN, 即 Virtual Extensible LAN (虚拟可扩展局域网)，是 Linux 本身支持的一网种网络虚拟化技术。VXLAN 可以完全在内核态实现封装和解封装工作，从而通过 “隧道” 机制，构建出覆盖网络 (Overlay Network)

基于三层的” 二层 “通信，层即 vxlan 包封装在 udp 数据包中，要求 udp 在 k8s 节点间三层可达；二层即 vxlan 封包的源 mac 地址和目的 mac 地址是自己的 vxlan 设备 mac 和对端 vxlan 设备 mac 实现通讯。

数据包封包：封包，在 vxlan 设备上将 pod 发来的数据包源、目的 mac 替换为本机 vxlan 网卡和对端节点 vxlan 网卡的 mac。外层 udp 目的 ip 地址根据路由和对端 vxlan 的 mac fdb 表获取

优势：只要 k8s 节点间三层互通，可以跨网段，对主机网关路由没有特殊要求。各个 node 节点通过 vxlan 设备实现基于三层的” 二层” 互通，三层即 vxlan 包封装在 udp 数据包中，要求 udp 在 k8s 节点间三层可达；二层即 vxlan 封包的源 mac 地址和目的 mac 地址是自己的 vxlan 设备 mac 和对端 vxlan 设备 mac

缺点：需要进行 vxan 的数据包封包和解包会存在一定的性能损耗

Calico 配置开启 VXLAN

#Enable IPIP
-name:CALICO IPV4POOL IPIP
-value:"Never"
#Enable or Disable VXLAN on the default IP pool.
-name:CALICO IPV4POOL VXLAN
-value:"Always"
#Enable or Disable VXLAN on the default IPv6 IP pool.
-name:CALICO IPV6POOL VXLAN
-value:"Always"


#calico_backend:"bird"
calico_backend:"vxlan"

#注释存活探测和就绪探测
#--bird-live
#--bird-ready

Calico 网络模式 --IPIP：

Linux 原生内核支持

IPIP 隧道的工作原理是将源主机的 IP 数据包封装在一个新的 P 数据包中，新的 IP 数据包的目的地址是隧道的另一端。在隧道的另一端，接收方将解封装原始 IP 数据包，并将其传递到目标主机。IPIP 隧道可以在不同的网络之间建立连接，例如在 IPV4 网络和 IPV6 网络之间建立连接。

数据包封包：封包，在 tunl0 设备上将 pod 发来的数据包的 mac 层去掉，留下 ip 层封包。
外层数据包目的 ip 地址根据路由得到。
优点：只要 k8s 节点间三层互通，可以跨网段，对主机网关路由没有特殊要求。
缺点：需要进行 IPIP 的数据包封包和解包会存在一定的性能损耗

Calico 配置开启 IPIP

#Enable IPIP
-name:CALICO IPV4POOL IPIP
-value:"Always"
#Enable or Disable VXLAN on the default IP pool.
-name:CALICO IPV4POOL VXLAN
-value:"Never"
#Enable or Disable VXLAN on the default IPv6 IP pool.
-name:CALICO IPV6POOL VXLAN
-value:"Never"

Calico 网络模式 --BGP：

边界网关协议 (Border Gateway Protocol,.BGP) 是互联网上一个核心的去中心化自治路由协议。它通过维护 IP 路由表或‘前缀 ' 表来实现自治系统 (AS) 之间的可达性，属于量路由协议。BGP 不使用传统的内部网关协议 (IGP) 的指标，而使用基于路径、网络策略或规则集来决定路由。因此，它更适合被称为矢量性协议，而不是路由协议。BGP，通俗的讲就是讲接入到机房的多条线路（如电信、联通，移动等）融合为一体，实现多线单 IP,BGP 机房的优点：服务器只需要设置一个 IP 地址，最佳访问路由是由网络上的骨干路由器根据路由跳数与其它技术指标来确定的，不会占用服务器的任何系统。

数据包封包：不需要进行数据包封包

优点：不用封包解包，通过 BGP 协议可实现 pod 网络在主机间的三层可达

缺点：跨网段时，配置较为复杂网络要求较高，主机网关路由也需要充当 BGP Speaker。.

Calico 配置开启 BGP

#Enable IPIP
-name:CALICO IPV4POOL IPIP
-value:"Off"
#Enable or Disable VXLAN on the default IP pool.
-name:CALICO IPV4POOL VXLAN
-value:"Never"
#Enable or Disable VXLAN on the default IPv6 IP pool.
-name:CALICO IPV6POOL VXLAN
-value:"Never"

Service#

在 Kubernetes 集群中，每个 Node 运行一个kube-proy进程。kube-proxy负责为 Service 实现了一种 VIP (虚拟 IP) 的形式。

在 Kubernetes v1.0 版本，代理完全在 userspace。在 Kubernetes v1.1 版本，新增了 iptables 代理，但并不是默认的运行模式。从 Kubernetes v1.2 起，默认就是 iptables 代理。在 Kubernetes v1.8.0-beta.0 中，添加了 ipvs 代理。

userspace

kube-proxy：

监听 APISERVER 将 Service 的变化修改本地的 iptables 规则
代理当前节点的 pod 用户请求

Iptables

kube-proxy：

监听 APISERVER 将 Service 的变化修改本地的 iptables 规则

相对于 userspace 方式，kube-proxy 功能解耦压力更小

ipvs

kube-proxy：

监听 APISERVER 将 Service 的变化修改本地的 ipvs 规则

Secret#

Kubernetes 通过仅仅将 Secret 分发到需要访问 Secret 的 Pod 所在的机器节点来保障其安全性。Secret 只会存储在节点的内存中，永不写入物理存储，这样从节点删除 secret 时就不需要擦除磁盘数据。

从 Kubernetes1.7 版本开始，etcd 会以加密形式存储 Secret,. 一定程度的保证了 Secret 安全性。

Secret 类型：

Downward API#

Downward API 是 Kubernetes 中的一个功能，它允许容器在运行时从 Kubernetes API 服务器获取有关它们自身的信息。这些信息可以作为容器内部的环境变量或文件注入到容器中，以便容器可以获取有关其运行环境的各种信息，如 Pod 名称、命名空间、标签等

提供容器元数据
动态配置
与 Kubernetes 环境集成

HELM#

Helm 是官方提供的类似于 YUM 的包管理器，是部署环境的流程封装。Helm 有两个重要的慨念：chart 和 release

Chart：是创建一个应用的信息集合，包括各种 Kubernetes 对象的配置模板、参数定义、依赖关系、文档说明等。chart 是应用部署的自包含逻辑单元。可以将 chart 想象成 apt、yum 中的软件安装包
Release：是 chart 的运行实例，代表了一个正在运行的应用。当 chart 被安装到 Kubernetes 集群，就生成一个 release。chart 能够多次安装到同一个集群，每次安装都是一个 release。
Hlelm cli：helm 客户端组件，负责和 kubernetes apiserver 通信
Repository：用于发布和存储 Chart 的仓库

下载安装#

下载 Helm

wget https://get.helm.sh/helm-v3.18.4-linux-amd64.tar.gz
tar -zxvf helm-v3.18.4-linux-amd64.tar.gz
cp -a linux-amd64/helm /usr/local/bin/
chmod a+x /usr/local/bin/helm
helm version

添加 chart 仓库国内源

helm repo add bitnami https://helm-charts.itboon.top/bitnami --force-update
helm repo update
#搜索仓库内容
helm search repo bitnami

安装 chart 示例#

#查看apache包配置
helm show values bitnami/apache
#安装apache
helm install bitnami/apache --generate-name
#查看
helm list -n default
kubectl get svc
kubectl get pod

#查看chart基本信息
helm show chart bitnami/apache
#查看chart所有信息
helm show all bitnami/apache

#删除版本
helm uninstall apache-1753181984
#保留历史版本
helm uninstall apache-1753181984 --keep-history

#查看该版本的信息
helm status apache-1753182488

拓展

#在当前仓库中搜索wordpress的cahrt包
helm search repo wordpress
#在官方仓库中搜索wordpress的cahrt包
helm search hub wordpress

#安装apache并指定名称为apache-1753234488
helm install apache-1753234488 bitnami/apache

安装自定义 chart

#查看apache包配置
helm show values bitnami/apache

#创建yaml文件，添加要修改的参数
vi apache.yml
service:
  type: NodePort

#覆盖配置参数，并安装apache
helm install -f apache.yml bitnami/apache --generate-name

除了使用 yaml 文件覆盖外，还可以使用--set：通过命令行的方式对指定项进行覆盖。如果同时使用两种方式，则--set中的值会被合并到--values中，但是--set中的值优先级更高，在--set中覆盖的内容会被被保存在 ConfigMap 中。可以通过helm get values <release-name>来查看指定 release 中--set设置的值。也可以通过运行helm upgrade并指定--reset-values字段来清除--set中设置的值。

--set的格式和限制

--set选项使用 0 或多个 name/value 对。最简单的用法类似于：--set name=value, 等价于如下 YAML 格式：

name:value

多个值使用逗号分割，因此--set a=b,c=d的 YAML 表示是：

a: b
c: d

支持更复杂的表达式。例如，--set outer.inner=value被转换成了：

outer:
  inner: value

列表使用花括号 ({}) 来表示，例如，--set name={a,b,c}被转换成了：

name:
  -a
  -b
  -c

某些 name/key 可以设置为 null 或者空数组，例如--set name=[],a=null

name: []
a: null

升级和回滚#

helm upgrade执行最小侵入式升级，只更新上次发布以来发生更改的内容。

#helm upgrade -f yaml文件 版本名 chart包
helm upgrade -f apache.yml apache-1753183272 bitnami/apache

版本回滚

#查看存在的版本
#helm history 版本名
helm history apache-1753183272
#执行回滚
#helm rollback 版本名 版本号
helm rollback apache-1753183272 1

创建自定义 chart 包#

#创建chart包
helm create test
#删除不需要的文件
#在templates中创建yaml资源清单
vi nodePort.yaml
############################
apiVersion: v1
kind: Service
metadata:
  name: myapp-test-202401110926-svc
  labels:
    app: myapp-test
spec:
  type: NodePort
  selector:
    app: myapp-test
  ports:
    - name: "80-80"
      protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 31111
############################
vi deplyment.yaml
############################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-test-202401110926-deploy
  labels:
    app: myapp-test
spec:
  replicas: 5
  selector:
    matchLabels:
      app: myapp-test
  template:
    metadata:
      labels:
        app: myapp-test
    spec:
      containers:
        - name: myapp
          image: wangyanglinux/myapp:v1.0
############################
#发布部署
helm install test test/

完整示例

vi templates/NOTES.txt
############################
1. 这是一个测试的 myapp chart
2. myapp release 名字：myapp-test-{{ now | date "20060102030405" }}-deploy
3. service 名字：myapp-test-{{ now | date "20060102030405" }}-svc
############################
vi templates/deplyment.yaml
############################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-test-{{ now | date "20060102030405" }}-deploy
  labels:
    app: myapp-test
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: myapp-test
  template:
    metadata:
      labels:
        app: myapp-test
    spec:
      containers:
        - name: myapp
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

############################
vi templates/service.yaml
############################
apiVersion: v1
kind: Service
metadata:
  name: myapp-test-{{ now | date "20060102030405" }}-svc
  labels:
    app: myapp-test
spec:
  type: {{ .Values.service.type | quote }}
  selector:
    app: myapp-test
  ports:
    - name: "80-80"
      protocol: TCP
      port: 80
      targetPort: 80
      {{- if eq .Values.service.type "NodePort" }}
      nodePort: {{ .Values.service.nodePort }}
      {{- end }}
############################
#与templates目录同一级
vi values.yaml
############################
replicaCount: 5
image:
  repository: wangyanglinux/myapp
  tag: "v1.0"  


service:
  type: NodePort
  nodePort: 32321  
############################

二进制高可用 Kubernetes 集群部署#

前言#

通过 5 台服务器使用二进制方式部署采用三主两从的高可用 Kubernetes 集群。

集群架构#

（1）基础环境

操作系统：Rocky Linux release 10.0

软件：Kubernetes-1.33.4、docker-28.3.3

（2）环境准备

主机名	IP	集群及组件角色
k8s-master01	192.168.0.111	master，api-server，control manager，scheduler，etcd， kubelet，kube-proxy，nginx
k8s-master02	192.168.0.112	master，api-server，control manager，scheduler，etcd， kubelet，kube-proxy，nginx
k8s-master03	192.168.0.113	master，api-server，control manager，scheduler，etcd， kubelet，kube-proxy，nginx
k8s-node01	192.168.0.114	worker，kubelet，kube-proxy，nginx
k8s-node02	192.168.0.115	worker，kubelet，kube-proxy，nginx

环境初始化#

（1）更换系统软件源，下载依赖软件

sed -e 's|^mirrorlist=|#mirrorlist=|g' \
    -e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g' \
    -i.bak \
    /etc/yum.repos.d/[Rr]ocky*.repo

dnf makecache
#下载依赖软件
yum install -y wget openssl gcc gcc-c++ zlib-devel openssl-devel make redhat-rpm-config

（2）重命名 hostname

hostnamectl set-hostname k8s-master01 && bash
hostnamectl set-hostname k8s-master02 && bash
hostnamectl set-hostname k8s-node01 && bash
hostnamectl set-hostname k8s-node02 && bash
hostnamectl set-hostname k8s-node03 && bash

（3）系统环境修改

#关闭firewalld防火墙
systemctl stop firewalld
systemctl disable firewalld
firewall-cmd --state

#安装iptables
yum install -y iptables-services
systemctl start iptables
iptables -F
systemctl enable iptables

# selinux永久关闭
setenforce 0
 sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
cat /etc/selinux/config

# swap永久关闭
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
cat /etc/fstab

# 设置时区
timedatectl set-timezone Asia/Shanghai
date

# 添加hosts
cat >> /etc/hosts << EOF
192.168.0.111 k8s-master01
192.168.0.112 k8s-master02
192.168.0.113 k8s-master03
192.168.0.114 k8s-node01
192.168.0.115 k8s-node02
EOF
#查看
cat /etc/hosts

（4）安装 ipvs

# 安装ipvs
yum -y install ipvsadm sysstat conntrack libseccomp

# 开启路由转发
echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf
sysctl -p

#ipvs加载模块
cat >> /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

systemctl restart systemd-modules-load.service

lsmod | grep -e ip_vs -e nf_conntrack

（5）排除 calico 网卡被 NetworkManager 所管理

# 排除 calico 网卡被 NetworkManager 所管理
cat > /etc/NetworkManager/conf.d/calico.conf << EOF 
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*
EOF

systemctl restart NetworkManager

（6）配置时间同步服务器

#修改k8s-master01上的chrony配置文件
sed -i -e 's/2\.rocky\.pool\.ntp\.org/ntp.aliyun.com/g' -e 's/#allow 192\.168\.0\.0\/16/allow 192.168.0.0\/24/g' -e 's/#local stratum 10/local stratum 10/g' /etc/chrony.conf

#修改k8s-master02上的chrony配置文件
sed -i -e 's/2\.rocky\.pool\.ntp\.org/ntp.aliyun.com/g' -e 's/#allow 192\.168\.0\.0\/16/allow 192.168.0.0\/24/g' -e 's/#local stratum 10/local stratum 11/g' /etc/chrony.conf

#修改k8s-master03上的chrony配置文件
sed -i -e 's/2\.rocky\.pool\.ntp\.org/ntp.aliyun.com/g' -e 's/#allow 192\.168\.0\.0\/16/allow 192.168.0.0\/24/g' -e 's/#local stratum 10/local stratum 12/g' /etc/chrony.conf

#修改k8s-node01、k8s-node02、k8s-node03上的chrony配置文件
sed -i 's/^pool 2\.rocky\.pool\.ntp\.org iburst$/pool 192.168.0.111 iburst\
pool 192.168.0.112 iburst\
pool 192.168.0.113 iburst/g' /etc/chrony.conf

#重启chronyd
systemctl restart chronyd

#验证
chronyc sources -v

（7）设置进程可打开的最大文件数

# 配置 ulimit
ulimit -SHn 65535

cat >> /etc/security/limits.conf << EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimitedd
EOF

（8）修改内核参数

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
EOF

sysctl --system

安装 docker#

（1）使用脚本安装 docker

bash <(curl -sSL https://linuxmirrors.cn/docker.sh)

（2）Docker 配置修改

cat >/etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": [
   "http://hub-mirror.c.163.com",
   "https://hub.rat.dev",
   "https://docker.mirrors.ustc.edu.cn",
   "https://docker.1panel.live",
   "https://docker.m.daocloud.io",
   "https://docker.1ms.run"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/data/dockerData"
}
EOF

systemctl daemon-reload
systemctl restart docker

安装 cri-dockerd#

（1）下载安装 cri-dockerd

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.18/cri-dockerd-0.3.18.amd64.tgz

tar xvf cri-dockerd-*.amd64.tgz 
cp -r cri-dockerd/*  /usr/bin/
chmod +x /usr/bin/cri-dockerd

（2）添加 cri-docker 服务配置文件

cat >  /usr/lib/systemd/system/cri-docker.service <<EOF
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.10
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

（3）添加 cri-docker 的 socket 配置文件

cat > /usr/lib/systemd/system/cri-docker.socket <<EOF
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target
EOF

（4）启动 cri-dockerd，使得配置生效

systemctl daemon-reload
systemctl enable --now cri-docker.service
systemctl status cri-docker.service

安装 etcd 集群（master 节点）#

（1）下载 etcd 包并安装

wget https://github.com/etcd-io/etcd/releases/download/v3.6.4/etcd-v3.6.4-linux-amd64.tar.gz

tar -xf etcd-*.tar.gz
mv etcd-*/etcd /usr/local/bin/ && mv etcd-*/etcdctl /usr/local/bin/
ls /usr/local/bin/
etcdctl version

安装 Kubernetes 集群#

（1）下载 Kubernetes 二进制包并安装

wget https://dl.k8s.io/v1.33.2/kubernetes-server-linux-amd64.tar.gz

#在master节点执行
tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
#在node节点执行
tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,-proxy}

ls /usr/local/bin/
kubelet --version
# 所有节点执行，存放cni插件
mkdir -p /opt/cni/bin

生成相关证书（master 节点）#

安装 cfssl 证书管理工具#

wget https://hub.gitmirror.com/https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl-certinfo_1.6.5_linux_amd64 -O /usr/local/bin/cfssl-certinfo
wget https://hub.gitmirror.com/https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssljson_1.6.5_linux_amd64 -O /usr/local/bin/cfssljson
wget https://hub.gitmirror.com/https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl_1.6.5_linux_amd64 -O /usr/local/bin/cfssl -O /usr/local/bin/cfssl

#添加可执行权限，查看版本
chmod +x /usr/local/bin/cfssl*
cfssl version

生成 ETCD 证书#

mkdir -p /etc/etcd/ssl && cd /etc/etcd/ssl

#创建生成证书的配置文件
cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

#创建证书签发请求文件
cat > etcd-ca-csr.json  << EOF 
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

签发 ETCD 的 CA 证书和密钥

cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

创建用于生成 ETCD 的服务端证书的配置文件

cat > etcd-csr.json << EOF 
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

签发 ETCD 的服务端证书

cfssl gencert    -ca=/etc/etcd/ssl/etcd-ca.pem    -ca-key=/etc/etcd/ssl/etcd-ca-key.pem    -config=ca-config.json -hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.0.111,192.168.0.112,192.168.0.113  -profile=kubernetes    etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

生成 Kubernetes 证书#

mkdir -p /etc/kubernetes/pki && cd /etc/kubernetes/pki
#创建证书签发请求文件
cat > ca-csr.json   << EOF 
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

签发 Kubernetes 的 CA 证书和密钥

cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca

生成 ApiServer 证书#

#创建证书签发请求文件
cat > apiserver-csr.json << EOF 
{
  "CN": "kube-apiserver",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

#创建生成证书的配置文件
cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

签发 ApiServer 的 CA 证书和密钥

cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem  -config=ca-config.json -hostname=10.96.0.1,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.0.111,192.168.0.112,192.168.0.113,192.168.0.114,192.168.0.115,192.168.0.116,192.168.0.117,192.168.0.118,192.168.0.119,192.168.0.120  -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

10.96.0.1 是 Kubernetes 的 service 的默认地址

kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local 是 Kubernetes 的默认解析域名

生成 ApiServer 聚合证书#

#创建证书签发请求文件
cat > front-proxy-ca-csr.json  << EOF 
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  },
  "ca": {
    "expiry": "876000h"
  }
}
EOF

签发 ApiServer 聚合证书和密钥

cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

生成 ApiServer 聚合证书的客户端证书#

#创建证书签发请求文件
cat > front-proxy-client-csr.json  << EOF 
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF

签发 ApiServer 聚合证书的客户端证书和密钥

cfssl gencert  \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem   \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   \
-config=ca-config.json   \
-profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client

生成 controller-manager 证书#

#创建证书签发请求文件
cat > manager-csr.json << EOF 
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

签发 controller-manager 证书和密钥

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager

生成 controller-manager 专用的 kubeconfig 配置文件

kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://127.0.0.1:8443 \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \
    --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
    --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
    --embed-certs=true \
    --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager@kubernetes \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

生成 kube-scheduler 证书#

#创建证书签发请求文件
cat > scheduler-csr.json << EOF 
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

签发 kube-scheduler 证书和密钥

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

生成 kube-scheduler 专用的 kubeconfig 配置文件

kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://127.0.0.1:8443 \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
     --client-certificate=/etc/kubernetes/pki/scheduler.pem \
     --client-key=/etc/kubernetes/pki/scheduler-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-context system:kube-scheduler@kubernetes \
     --cluster=kubernetes \
     --user=system:kube-scheduler \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config use-context system:kube-scheduler@kubernetes \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

生成 admin 证书#

#创建证书签发请求文件
cat > admin-csr.json << EOF 
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

签发 admin 证书和密钥

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

生成 admin 专用的 kubeconfig 配置文件

kubectl config set-cluster kubernetes     \
  --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  --embed-certs=true     \
  --server=https://127.0.0.1:8443     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config set-credentials kubernetes-admin  \
  --client-certificate=/etc/kubernetes/pki/admin.pem     \
  --client-key=/etc/kubernetes/pki/admin-key.pem     \
  --embed-certs=true     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config set-context kubernetes-admin@kubernetes    \
  --cluster=kubernetes     \
  --user=kubernetes-admin     \
  --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config use-context kubernetes-admin@kubernetes  --kubeconfig=/etc/kubernetes/admin.kubeconfig

生成 kube-proxy 证书#

#创建证书签发请求文件
cat > kube-proxy-csr.json  << EOF 
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-proxy",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

签发 kube-proxy 证书和密钥

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy

生成 kube-proxy 专用的 kubeconfig 配置文件

kubectl config set-cluster kubernetes     \
  --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  --embed-certs=true     \
  --server=https://127.0.0.1:8443     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy  \
  --client-certificate=/etc/kubernetes/pki/kube-proxy.pem     \
  --client-key=/etc/kubernetes/pki/kube-proxy-key.pem     \
  --embed-certs=true     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config set-context kube-proxy@kubernetes    \
  --cluster=kubernetes     \
  --user=kube-proxy     \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config use-context kube-proxy@kubernetes  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

创建 ServiceAccount 加密密钥#

openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

添加组件配置，启动服务#

ETCD 组件（master 节点）#

（1）k8s-master01 配置文件

cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.0.111:2380'
listen-client-urls: 'https://192.168.0.111:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.0.111:2380'
advertise-client-urls: 'https://192.168.0.111:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.0.111:2380,k8s-master02=https://192.168.0.112:2380,k8s-master03=https://192.168.0.113:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

（2）k8s-master02 配置文件

cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.0.112:2380'
listen-client-urls: 'https://192.168.0.112:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.0.112:2380'
advertise-client-urls: 'https://192.168.0.112:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.0.111:2380,k8s-master02=https://192.168.0.112:2380,k8s-master03=https://192.168.0.113:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

（3）k8s-master03 配置文件

cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.0.113:2380'
listen-client-urls: 'https://192.168.0.113:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.0.113:2380'
advertise-client-urls: 'https://192.168.0.113:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.0.111:2380,k8s-master02=https://192.168.0.112:2380,k8s-master03=https://192.168.0.113:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

（4）创建 etcd 服务启动配置文件

cat > /usr/lib/systemd/system/etcd.service << EOF

[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

EOF

（5）启动 etcd 服务

mkdir -p /etc/kubernetes/pki/etcd
ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd.service

（6）etcd 集群的健康状态

export ETCDCTL_API=3
etcdctl --endpoints="192.168.0.111:2379,192.168.0.112:2379,192.168.0.113:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table

+--------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+--------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| 192.168.0.111:2379 | 9c35553b47538310 | 3.6.4 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 3 | 6 | 6 | | | false |
| 192.168.0.112:2379 | 545bae002651f913 | 3.6.4 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 7 | 7 | | | false |
| 192.168.0.113:2379 | d7497b3a31d15f9e | 3.6.4 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 7 | 7 | | | false |
+--------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
#将etcd相关防火墙策略保存，防止重启后etcd服务无法启动
service iptables save
如果出现重启服务器后，etcd 集群服务无法启动的情况。清空防火墙规则，启动 etcd 服务并保存策略即可。

安装 Nginx 配置高可用#

（1）下载安装 Nginx

wget https://nginx.org/download/nginx-1.28.0.tar.gz
tar xvf nginx-1.28.0.tar.gz
cd nginx-1.28.0
#编译安装，其中--with-stream为启用四层代理
./configure --with-stream --without-http --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
make && make install

（2）创建配置文件

cat > /usr/local/nginx/conf/kube-nginx.conf <<EOF
worker_processes 1;
events {
    worker_connections  1024;
}
stream {
    upstream backend {
    	least_conn;
        hash $remote_addr consistent;
        server 192.168.0.111:6443        max_fails=3 fail_timeout=30s;
        server 192.168.0.112:6443        max_fails=3 fail_timeout=30s;
        server 192.168.0.113:6443        max_fails=3 fail_timeout=30s;
    }
    server {
        listen 127.0.0.1:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF

（6）添加 nignx 服务

cat > /etc/systemd/system/kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
ExecStartPre=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -t
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx
ExecReload=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target
EOF

（7）启动服务

systemctl daemon-reload
systemctl enable --now kube-nginx.service
systemctl status kube-nginx.service

ApiServer 组件#

#所有节点创建所需目录
mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

（1）k8s-master01 节点添加 apiserver 服务

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.0.111 \\
      --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.0.111:2379,https://192.168.0.112:2379,https://192.168.0.113:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF

（2）k8s-master02 节点添加 apiserver 服务

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.0.112 \\
      --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.0.111:2379,https://192.168.0.112:2379,https://192.168.0.113:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
      --kubelet-client-key=/etc/kuberntes/pki/apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true


Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF

（3）k8s-master03 节点添加 apiserver 服务

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.0.113 \\
      --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.0.111:2379,https://192.168.0.112:2379,https://192.168.0.113:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true


Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF

（4）启动 kube-apiserver 服务

systemctl daemon-reload 
systemctl enable --now kube-apiserver
systemctl status kube-apiserver

controller-manager 组件#

所有 master 节点添加 controller-manager 服务

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
      --v=2 \\
      --bind-address=0.0.0.0 \\
      --root-ca-file=/etc/kubernetes/pki/ca.pem \\
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \\
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \\
      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \\
      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \\
      --leader-elect=true \\
      --use-service-account-credentials=true \\
      --node-monitor-grace-period=40s \\
      --node-monitor-period=5s \\
      --controllers=*,bootstrapsigner,tokencleaner \\
      --allocate-node-cidrs=true \\
      --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \\
      --cluster-cidr=172.16.0.0/12,fc00:2222::/112 \\
      --node-cidr-mask-size-ipv4=24 \\
      --node-cidr-mask-size-ipv6=120 \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem 

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

EOF

启动 controller-manager 服务

systemctl daemon-reload
systemctl enable --now kube-controller-manager
systemctl  status kube-controller-manager

kube-scheduler 组件（master 节点）#

所有 master 节点添加 kube-scheduler 服务

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
      --v=2 \\
      --bind-address=0.0.0.0 \\
      --leader-elect=true \\
      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

EOF

启动 kube-scheduler 服务

systemctl daemon-reload
systemctl enable --now kube-scheduler
systemctl status kube-scheduler

TLS Bootstrapping 配置（master01 节点，用于自动签发证书）#

kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     --server=https://127.0.0.1:8443     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-credentials tls-bootstrap-token-user     \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes     \
--cluster=kubernetes     \
--user=tls-bootstrap-token-user     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

# token的位置在bootstrap.secret.yaml，如果修改的话到这个文件修改
mkdir -p /root/.kube && cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

查看集群状态

kubectl get cs

创建 bootstrap-secret 授权文件

cat > bootstrap-secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: "c8ad9c"
  token-secret: "2e4d610cf3e7426e"
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-certificate-rotation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver
EOF

执行 bootstrap-secret 授权文件

kubectl apply -f bootstrap-secret.yaml

将证书文件复制到其他节点

cd /etc/kubernetes/
for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02; do ssh $NODE mkdir -p /etc/kubernetes/pki; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig kube-proxy.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done

Kubelet 组件（所有节点）#

（1）添加 kubelet 服务

cat > /usr/lib/systemd/system/kubelet.service << EOF

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig  \\
    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
    --config=/etc/kubernetes/kubelet-conf.yml \\
    --container-runtime-endpoint=unix:///run/cri-dockerd.sock  \\
    --node-labels=node.kubernetes.io/node= 


[Install]
WantedBy=multi-user.target
EOF

（2）创建 kubelet 配置文件

cat > /etc/kubernetes/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF

（3）启动 kubelet 服务

systemctl daemon-reload
systemctl enable --now kubelet
systemctl status kubelet

kube-proxy 组件（所有节点）#

（1）将证书发送至其他节点

for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; done

（2）添加 kube-proxy 服务

cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy.yaml \\
  --cluster-cidr=172.16.0.0/12,fc00:2222::/112 \\
  --v=2
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

EOF

（3）创建 kube-proxy 配置文件

cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12,fc00:2222::/112
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
EOF

（4）启动 kube-proxy 服务

systemctl daemon-reload
systemctl enable --now kube-proxy.service
systemctl status kube-proxy.service

安装 Calico 网络插件#

（1）下载安装 Calico 插件

wget https://hub.gitmirror.com/https://raw.githubusercontent.com/projectcalico/calico/v3.30.2/manifests/calico.yaml 


#取消注释并修改pod网端
- name: CALICO_IPV4POOL_CIDR
   value: "172.16.0.0/12"

#更改容器镜像源
sed -i "s#docker.io/calico/#m.daocloud.io/docker.io/calico/#g" calico.yaml
#执行安装
kubectl apply -f calico.yaml

安装 CoreDNS#

（1）下载安装 Helm

wget https://get.helm.sh/helm-v3.18.4-linux-amd64.tar.gz
tar -zxvf helm-v3.18.4-linux-amd64.tar.gz
cp -a linux-amd64/helm /usr/local/bin/
chmod a+x /usr/local/bin/helm
helm version

（2）使用 Helm 安装 CoreDNS

helm repo add coredns https://coredns.github.io/helm
helm pull coredns/coredns
tar xvf coredns-*.tgz
cd coredns/

（3）修改 values.yml 文件

clusterIP: "10.96.0.10"

#更换镜像源
sed -i  's|coredns/coredns|m.daocloud.io/docker.io/coredns/coredns|g' values.yaml
 sed -i "s|registry.k8s.io/cpa/cluster-proportional-autoscaler|m.daocloud.io/registry.k8s.io/cpa/cluster-proportional-autoscaler|g" values.yaml

（4）安装

helm install  coredns ./ -n kube-system
#查看安装情况
kubectl get pod -n kube-system

安装 Metrics Server#

wget https://hub.gitmirror.com/https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml

#修改配置
      - args:
        - --cert-dir=/tmp
        - --secure-port=10250
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
        - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
        - --requestheader-username-headers=X-Remote-User
        - --requestheader-group-headers=X-Remote-Group
        - --requestheader-extra-headers-prefix=X-Remote-Extra-



        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
        - name: ca-ssl
          mountPath: /etc/kubernetes/pki

          

      volumes:
      - emptyDir: {}
        name: tmp-dir
      - name: ca-ssl
        hostPath:
          path: /etc/kubernetes/pki
          
          
#修改镜像源
sed -i "s#registry.k8s.io/#m.daocloud.io/registry.k8s.io/#g" components.yaml

#执行部署
kubectl apply -f components.yaml

#验证
 kubectl top pod -A

设置各节点角色标签和污点#

（1）设置 master 节点污点

kubectl taint node k8s-master01 node-role.kubernetes.io/master:NoSchedule
kubectl taint node k8s-master02 node-role.kubernetes.io/master:NoSchedule
kubectl taint node k8s-master03 node-role.kubernetes.io/master:NoSchedule

（2）设置各节点角色标签

#k8s-master01
kubectl label nodes k8s-master01 node-role.kubernetes.io/master=
kubectl label nodes k8s-master01 node-role.kubernetes.io/control-plane=
#k8s-master02
kubectl label nodes k8s-master02 node-role.kubernetes.io/master=
kubectl label nodes k8s-master02 node-role.kubernetes.io/control-plane=
#k8s-master03
kubectl label nodes k8s-master03 node-role.kubernetes.io/master=
kubectl label nodes k8s-master03 node-role.kubernetes.io/control-plane=
#k8s-node01
kubectl label nodes k8s-node01 node-role.kubernetes.io/worker=
#k8s-node02
kubectl label nodes k8s-node02 node-role.kubernetes.io/worker=

设置命令补全#

yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

验证高可用#

（1）验证 etcd 集群高可用

#验证etcd集群高可用
export ETCDCTL_API=3
etcdctl --endpoints="192.168.0.113:2379,192.168.0.112:2379,192.168.0.111:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table

（2）使用 etcdhelper 查询 etcd 中 k8s 的资源数据

#拉取etcdhelper源码文件
git clone --depth 1 https://hub.gitmirror.com/https://github.com/openshift/origin.git

#下载依赖软件
yum install -y go
#构建etcdhelper执行程序
cd origin/tools/etcdhelper
go build etcdhelper.go
mv etcdhelper /usr/local/bin/

#设定参数别名
echo 'alias ectl="etcdhelper -endpoint https://192.168.0.111:2379 -cacert /etc/etcd/ssl/etcd-ca.pem -key /etc/etcd/ssl/etcd-key.pem -cert /etc/etcd/ssl/etcd.pem"' >> ~/.bashrc

#执行配置生效
source ~/.bashrc

#查看etcd数据
ectl ls
# 查看当前的 schedule 主节点
ectl get /registry/leases/kube-system/kube-scheduler
# 查看当前的 controllermanager 主节点
ectl get /registry/leases/kube-system/kube-controller-manager

部署 deployment 验证#

vi test.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test01
spec:
  replicas: 6
  selector:
    matchLabels:
      app: test01  
  template:
    metadata:
      labels:
        app: test01 
    spec:
      containers:
      - name: test01
        image: nginx
        ports:
        - containerPort: 80  

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test02
spec:
  replicas: 5
  selector:
    matchLabels:
      app: test02
  template:
    metadata:
      labels:
        app: test02
    spec:
      nodeName: k8s-node02
      containers:
      - name: test02
        image: nginx
        ports:
        - containerPort: 80  

---

apiVersion: v1
kind: Service
metadata:
  name: test01
spec:
  type: NodePort  
  selector:
    app: test01
  ports:
  - name: http
    port: 80       
    targetPort: 80  

---

apiVersion: v1
kind: Service
metadata:
  name: test02
spec:
  type: NodePort  
  selector:
    app: test02
  ports:
  - name: http
    port: 80       
    targetPort: 80 
    
    
#部署
kubectl apply -f test.yml
#查看
kubectl  get pod,svc,deployments -o wide
#在浏览器访问查看nginx访问情况

本篇知识来源于 B 站视频 BV1PbeueyE8V