Ubuntu: 2020.4
Docker: 19.3
Kubernetes: v1.20.4
Rancher: v2.5.8
如果要使用rancher请先到官网查看不同的版本支持的k8s的版本
https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.5.8/
可以看到rancher v2.5.8 支持的k8s最新版是v1.20.4
https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG
找到k8s版本的changelog 查看k8s支持的docker版本
hostnamectl set-hostname k8s-01
vim /etc/hosts 192.168.70.20 k8s-01 192.168.70.21 k8s-02 192.168.70.22 k8s-03 添加多个ip解析
systemctl stop firewalld && systemctl disable firewalld
sudo apt-get install -y ipvsadm iptables wget vim net-tools
swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
cat > kubernetes.conf <<EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 net.ipv4.tcp_tw_recycle=0 vm.swappiness=0 # 禁止使用 swap 空间,只有当系统 OOM 时才允许使用它 vm.overcommit_memory=1 # 不检查物理内存是否够用 vm.panic_on_oom=0 # 开启 OOM fs.inotify.max_user_instances=8192 fs.inotify.max_user_watches=1048576 fs.file-max=52706963 fs.nr_open=52706963 net.ipv6.conf.all.disable_ipv6=1 net.netfilter.nf_conntrack_max=2310720 EOF cp kubernetes.conf /etc/sysctl.d/kubernetes.conf sysctl -p /etc/sysctl.d/kubernetes.conf
保证多个机器的时间一致
# 设置系统时区为 中国/上海 timedatectl set-timezone Asia/Shanghai # 将当前的 UTC 时间写入硬件时钟 timedatectl set-local-rtc 0 # 重启依赖于系统时间的服务 systemctl restart rsyslog systemctl restart crond
mkdir -p /etc/systemd cat > journald.conf <<EOF [Journal] # 持久化保存到磁盘 Storage=persistent # 压缩历史日志 Compress=yes SyncIntervalSec=5m RateLimitInterval=30s RateLimitBurst=1000 # 最大占用空间 10G SystemMaxUse=10G # 单日志文件最大 200M SystemMaxFileSize=200M # 日志保存时间 2 周 MaxRetentionSec=2week # 不将日志转发到 syslog ForwardToSyslog=no EOF mv journald.conf /etc/systemd/
sudo apt-get update
sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg-agent \ software-properties-common
没有这一步apt update的时候会报错
curl -fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \ "deb [arch=amd64] https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/ \ $(lsb_release -cs) \ stable"
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
apt-cache madison docker-ce
5:20.10.73-0ubuntu-focal 就是<VERSION_STRING>
sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io
load balance采用官方推荐的HA proxy + keepalives策略
https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#options-for-software-load-balancing
haproxy使用docker部署 ,多个master节点都需要部署
https://registry.hub.docker.com/_/haproxy?tab=tags&page=1&ordering=last_updated
# /etc/haproxy/haproxy.cfg #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log /dev/log local0 log /dev/log local1 notice daemon #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 1 timeout http-request 10s timeout queue 20s timeout connect 5s timeout client 20s timeout server 20s timeout http-keep-alive 10s timeout check 10s #前页面配置 访问/dbs listen admin_stats bind 0.0.0.0:7050 mode http stats uri /dbs stats realm Global\ statistics stats auth admin:123456 #--------------------------------------------------------------------- # apiserver frontend which proxys to the control plane nodes #--------------------------------------------------------------------- frontend apiserver bind *:${APISERVER_DEST_PORT} mode tcp option tcplog default_backend apiserver #--------------------------------------------------------------------- # round robin balancing for apiserver # 注意 如果在集群刚部署时apiserver下面只留一个server 全部部署好后在加上 #--------------------------------------------------------------------- backend apiserver mode tcp option ssl-hello-chk balance roundrobin server k8s-01 ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} server k8s-02 ${HOST2_ADDRESS}:${APISERVER_SRC_PORT} server k8s-03 ${HOST3_ADDRESS}:${APISERVER_SRC_PORT}
该脚本默认haproxy.cfg在/opt/k8s/haproxy 目录下
vim start-haproxy.sh #!/bin/bash HOST1_ADDRESS=192.168.70.20 HOST2_ADDRESS=192.168.70.21 HOST3_ADDRESS=192.168.70.22 APISERVER_SRC_PORT=6443 APISERVER_DEST_PORT=6444 docker run -d --restart=always --name HAProxy-K8S -p 6444:6444 -p 7050:7050 \ -e HOST1_ADDRESS=$HOST1_ADDRESS \ -e HOST2_ADDRESS=$HOST2_ADDRESS \ -e HOST3_ADDRESS=$HOST3_ADDRESS \ -e APISERVER_DEST_PORT=$APISERVER_DEST_PORT \ -e APISERVER_SRC_PORT=$APISERVER_SRC_PORT \ -v /opt/k8s/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg \ haproxy
chmod +x start-haproxy.sh sudo ./start-haproxy.sh
浏览器访问ip:7050/dbc 用户名 admin 密码 123456
配置: https://blog.csdn.net/get_set/article/details/107998958
所有master节点都需要安装
sudo apt install -y keepalived
cat >/etc/keepalived/keepalived.conf<<"EOF" ! Configuration File for keepalived global_defs { #可以填写你的hostname router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" #间隔 interval 3 #权重 weight -2 fall 3 } vrrp_instance VI_1 { # 多个master 只有一个配置为MASTER其他为BACKUP state MASTER # ifconfig查看你的当前网卡 配置在这里 interface eth0 #当前ip地址 mcast_src_ip 192.168.1.151 #保证每个机器上的virtual_router_id不一样 virtual_router_id 51 priority 101 advert_int 2 authentication { auth_type PASS auth_pass 123456 } #配置为和你当前网段相同的ip virtual_ipaddress { 192.168.1.160 } #这部分在集群没有打起来之前需要先注释掉 track_script { chk_apiserver } } EOF
systemctl enable keepalived --now
ip a
在你当前网卡下找到你配置ip,如果能够看到代表配置生效,否则就要排查问题.
配置: https://www.keepalived.org/manpage.html
原理: https://www.cnblogs.com/rexcheny/p/10778567.html
sudo apt-add-repository "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - sudo apt-get update
最新版安装
sudo apt -y install kubeadm kubectl kubelet systemctl enable kubelet.service
安装指定版本
apt-cache madison kubeadm sudo apt -y install kubeadm=<VERSION_STRING> kubectl=<VERSION_STRING> kubelet=<VERSION_STRING> systemctl enable kubelet.service
kubeadm --kubernetes-version=v1.21.0 config images list
因为k8s.gcr.io需要翻墙,所以从阿里云下载,所有master节点都需要下载
docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.21.0 docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.21.0 docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.21.0 docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.21.0 docker pull registry.aliyuncs.com/google_containers/pause:3.2 docker pull registry.aliyuncs.com/google_containers/etcd:3.4.13-0 docker pull coredns/coredns:1.7.0
之所以从dockerhub上下载dns是因为aliyun下没有它的镜像,之后讲coredns的镜像重新打tag
docker images|grep coredns docker tag 镜像id docker pull registry.aliyuncs.com/google_containers/coredns:1.7.0
kubeadm config print init-defaults > kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: #修改为当前ip地址 advertiseAddress: 1.2.3.4 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock #修改为当前的hostname name: k8s-02 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes #默认配置下没有这个配置,如果是集群需要手动加上,否则不需要 #这个ip为keepalived配置的vip,端口为haproxy监听的端口 controlPlaneEndpoint: 192.168.70.100:6444 controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.20.4 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 #该位置也需要手动加上是fannel的配置 podSubnet: 10.244.0.0/16 scheduler: {} --- #下面的配置是启动ipvs的配置,每个版本都不一样 apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: ipvs
https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md
kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.log
启动成功会出下如下画面
这时一个master节点已经搭建成功,首先指定下列命令
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config root用户指定 export KUBECONFIG=/etc/kubernetes/admin.conf
其他机器加入到master,执行
在指定该命令之前,请确保你的第一个主节点上的haproxy代理的一个server kubeadm join 192.168.70.100:6444 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:db5964c163eaf5a280616ab0736dae7056293855a6d1811d78496e070bc94202 \ --control-plane --certificate-key 7f819f2d00cfdf75e8318a1dc40bc41dbf2540084742a5fa31c68d49316ff390
加入node节点执行
kubeadm join 192.168.70.100:6444 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:db5964c163eaf5a280616ab0736dae7056293855a6d1811d78496e070bc94202
所有节点全部加入以后执行
kubectl get nodes
可以看到集群是NotReady的状态,需要部署flannel
官网: https://github.com/flannel-io/flannel
部署
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
网络原因可以先将kube-flannel.yml下载下来
kubectl apply -f kube-flannel.yml
部署完成后
kubectl get nodes
https://helm.sh/zh/docs/intro/install/
官网: https://kubernetes.github.io/ingress-nginx/deploy/
github: https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/index.md
执行Bare-metal下的安装命令
中文文档: http://docs.rancher.cn/rancher2.5/
安装: http://docs.rancher.cn/docs/rancher2.5/installation/install-rancher-on-k8s/_index
latest
: 建议在尝试新功能时使用。stable
: 建议在生产环境中使用。(推荐)alpha
: 未来版本的实验性预览。helm repo add rancher-<CHART_REPO> http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/<CHART_REPO>
kubectl create namespace cattle-system
创建自签名证书
http://docs.rancher.cn/docs/rancher2.5/installation/resources/advanced/self-signed-ssl/_index/
#!/bin/bash -e help () { echo ' ================================================================ ' echo ' --ssl-domain: 生成ssl证书需要的主域名,如不指定则默认为www.rancher.local,如果是ip访问服务,则可忽略;' echo ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求,有时候需要使用ip去访问server,那么需要给ssl证书添加扩展IP,多个IP用逗号隔开;' echo ' --ssl-trusted-domain: 如果想多个域名访问,则添加扩展域名(SSL_TRUSTED_DOMAIN),多个扩展域名用逗号隔开;' echo ' --ssl-size: ssl加密位数,默认2048;' echo ' --ssl-cn: 国家代码(2个字母的代号),默认CN;' echo ' 使用示例:' echo ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ ' echo ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650' echo ' ================================================================' } case "$1" in -h|--help) help; exit;; esac if [[ $1 == '' ]];then help; exit; fi CMDOPTS="$*" for OPTS in $CMDOPTS; do key=$(echo ${OPTS} | awk -F"=" '{print $1}' ) value=$(echo ${OPTS} | awk -F"=" '{print $2}' ) case "$key" in --ssl-domain) SSL_DOMAIN=$value ;; --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;; --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;; --ssl-size) SSL_SIZE=$value ;; --ssl-date) SSL_DATE=$value ;; --ca-date) CA_DATE=$value ;; --ssl-cn) CN=$value ;; esac done # CA相关配置 CA_DATE=${CA_DATE:-3650} CA_KEY=${CA_KEY:-cakey.pem} CA_CERT=${CA_CERT:-cacerts.pem} CA_DOMAIN=cattle-ca # ssl相关配置 SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf} SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'} SSL_DATE=${SSL_DATE:-3650} SSL_SIZE=${SSL_SIZE:-2048} ## 国家代码(2个字母的代号),默认CN; CN=${CN:-CN} SSL_KEY=$SSL_DOMAIN.key SSL_CSR=$SSL_DOMAIN.csr SSL_CERT=$SSL_DOMAIN.crt echo -e "\033[32m ---------------------------- \033[0m" echo -e "\033[32m | 生成 SSL Cert | \033[0m" echo -e "\033[32m ---------------------------- \033[0m" if [[ -e ./${CA_KEY} ]]; then echo -e "\033[32m ====> 1. 发现已存在CA私钥,备份"${CA_KEY}"为"${CA_KEY}"-bak,然后重新创建 \033[0m" mv ${CA_KEY} "${CA_KEY}"-bak openssl genrsa -out ${CA_KEY} ${SSL_SIZE} else echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m" openssl genrsa -out ${CA_KEY} ${SSL_SIZE} fi if [[ -e ./${CA_CERT} ]]; then echo -e "\033[32m ====> 2. 发现已存在CA证书,先备份"${CA_CERT}"为"${CA_CERT}"-bak,然后重新创建 \033[0m" mv ${CA_CERT} "${CA_CERT}"-bak openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" else echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m" openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" fi echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m" cat > ${SSL_CONFIG} <<EOM [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOM if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then cat >> ${SSL_CONFIG} <<EOM subjectAltName = @alt_names [alt_names] EOM IFS="," dns=(${SSL_TRUSTED_DOMAIN}) dns+=(${SSL_DOMAIN}) for i in "${!dns[@]}"; do echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG} done if [[ -n ${SSL_TRUSTED_IP} ]]; then ip=(${SSL_TRUSTED_IP}) for i in "${!ip[@]}"; do echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG} done fi fi echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m" openssl genrsa -out ${SSL_KEY} ${SSL_SIZE} echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m" openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG} echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m" openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \ -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \ -days ${SSL_DATE} -extensions v3_req \ -extfile ${SSL_CONFIG} echo -e "\033[32m ====> 7. 证书制作完成 \033[0m" echo echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m" echo "----------------------------------------------------------" echo "ca_key: |" cat $CA_KEY | sed 's/^/ /' echo echo "ca_cert: |" cat $CA_CERT | sed 's/^/ /' echo echo "ssl_key: |" cat $SSL_KEY | sed 's/^/ /' echo echo "ssl_csr: |" cat $SSL_CSR | sed 's/^/ /' echo echo "ssl_cert: |" cat $SSL_CERT | sed 's/^/ /' echo echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m" cat ${CA_CERT} >> ${SSL_CERT} echo "ssl_cert: |" cat $SSL_CERT | sed 's/^/ /' echo echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m" echo "cp ${SSL_DOMAIN}.key tls.key" cp ${SSL_DOMAIN}.key tls.key echo "cp ${SSL_DOMAIN}.crt tls.crt" cp ${SSL_DOMAIN}.crt tls.crt
/start.sh --ssl-domain=rancher.test.cn --ssl-trusted-domain=rancher.test.cn --ssl-trusted-ip=192.168.20.101,192.168.21.30,192.168.21.31 --ssl-size=2048 --ssl-date=3650
kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=tls.crt --key=tls.key kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem=./cacerts.pem
这里的hostname需要和上面生成的证书域名保持一致
helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.test.cn --set ingress.tls.source=secret --set privateCA=true
kubectl -n cattle-system rollout status deploy/rancher
若出现下面信息(successfully rolled out)则表示成功,否则继续等待:
deployment "rancher" successfully rolled out
检查 deployment 的状态:
kubectl -n cattle-system get deploy rancher
若出现下面信息则表示安装成功
NAME READY UP-TO-DATE AVAILABLE AGE rancher 3/3 3 3 2d1h
正常情况下在电脑下修改host 配置域名和ip 通过域名访问即可
https://域名
kubectl get svc -n cattle-system
从图上可以rancher的port 是 80 和443 端口,但是它的类型是ClusterIP.这代表这个ip和端口都是虚拟的我们是不能直接访问的,需要通过ingress暴漏一个NodePort的端口让我们访问
vim service-nodeport.yaml
apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 protocol: TCP - name: https port: 443 targetPort: 443 protocol: TCP selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ---
kubectl apply -f service-nodeport.yaml
kubectl get svc -n ingress-nginx
如上图所示,该服务的类型是NodePort ,我们可以通过30001访问80端口
https://域名:30002访问