HPA官方文档 :https://kubernetes.io/zh/docs/tasks/run-application/horizontal-pod-autoscale/
apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx namespace: hpa spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx resources: requests: cpu: 200m memory: 100Mi --- apiVersion: v1 kind: Service metadata: name: nginx namespace: hpa spec: type: NodePort ports: - port: 80 targetPort: 80 selector: app: nginx
# kubectl get po -n hpa NAME READY STATUS RESTARTS AGE nginx-5c87768612-48b4v 1/1 Running 0 8m38s nginx-5c87768612-kfpkq 1/1 Running 0 8m38s
apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: nginx namespace: hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx minReplicas: 1 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50
# kubectl get hpa -n hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx Deployment/nginx 0%/50% 1 10 2 50s
# 执行压测命令 # ab -c 1000 -n 100000000 http://127.0.0.1:30792/ This is ApacheBench, Version 2.3 <$Revision: 1843412 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient)
# 观察变化 # kubectl get hpa -n hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx Deployment/nginx 303%/50% 1 10 7 12m # kubectl get po -n hpa NAME READY STATUS RESTARTS AGE pod/nginx-5c87768612-6b4sl 1/1 Running 0 85s pod/nginx-5c87768612-99mjb 1/1 Running 0 69s pod/nginx-5c87768612-cls7r 1/1 Running 0 85s pod/nginx-5c87768612-hhdr7 1/1 Running 0 69s pod/nginx-5c87768612-jj744 1/1 Running 0 85s pod/nginx-5c87768612-kfpkq 1/1 Running 0 27m pod/nginx-5c87768612-xb94x 1/1 Running 0 69s
# kubectl get hpa -n hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx Deployment/nginx 20%/50% 1 10 7 16m ---N分钟后--- # kubectl get hpa -n hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx Deployment/nginx 0%/50% 1 10 7 18m ---再过N分钟后--- # kubectl get po -n hpa NAME READY STATUS RESTARTS AGE nginx-5c87768612-jj744 1/1 Running 0 11m
VPA项目托管地址 :https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler
参考博文 :https://www.jianshu.com/p/94ea8bee433e
# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml
- name: metrics-server image: scofield/metrics-server:v0.3.7 imagePullPolicy: IfNotPresent args: - --cert-dir=/tmp - --secure-port=4443 - /metrics-server - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP
# kubectl apply -f components.yaml # kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE metrics-server-7947cb98b6-xw6b8 1/1 Running 0 10m # kubectl top nodes
# git clone https://github.com/kubernetes/autoscaler.git
# cd autoscaler/vertical-pod-autoscaler # ./hack/vpa-up.sh Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created clusterrole.rbac.authorization.k8s.io/system:metrics-reader created clusterrole.rbac.authorization.k8s.io/system:vpa-actor created clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created clusterrole.rbac.authorization.k8s.io/system:evictioner created clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictionter-binding created serviceaccount/vpa-admission-controller created clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created serviceaccount/vpa-updater created deployment.apps/vpa-updater created serviceaccount/vpa-recommender created deployment.apps/vpa-recommender created Generating certs for the VPA Admission Controller in /tmp/vpa-certs. Generating RSA private key, 2048 bit long modulus (2 primes) ............................................................................+++++ .+++++ e is 65537 (0x010001) Generating RSA private key, 2048 bit long modulus (2 primes) ............+++++ ...........................................................................+++++ e is 65537 (0x010001) Signature ok subject=CN = vpa-webhook.kube-system.svc Getting CA Private Key Uploading certs to the cluster. secret/vpa-tls-certs created Deleting /tmp/vpa-certs. deployment.apps/vpa-admission-controller created service/vpa-webhook created
# 可以看到metrics-server和vpa都已经正常运行了 # kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE metrics-server-7947cb98b6-xw6b8 1/1 Running 0 46m vpa-admission-controller-7d87559549-g77h9 1/1 Running 0 10m vpa-recommender-84bf7fb9db-65669 1/1 Running 0 10m vpa-updater-79cc46c7bb-5p889 1/1 Running 0 10m
apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx namespace: vpa spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx resources: requests: cpu: 100m memory: 250Mi
# cat nginx-vpa-ingress.yaml apiVersion: v1 kind: Service metadata: name: nginx namespace: vpa spec: type: NodePort ports: - port: 80 targetPort: 80 selector: app: nginx # kubectl get svc -n vpa NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx NodePort 10.97.250.131 <none> 80:32621/TCP 55s
# cat nginx-vpa-demo.yaml apiVersion: autoscaling.k8s.io/v1beta2 kind: VerticalPodAutoscaler metadata: name: nginx-vpa namespace: vpa spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: nginx updatePolicy: updateMode: "Off" resourcePolicy: containerPolicies: - containerName: "nginx" minAllowed: cpu: "250m" memory: "100Mi" maxAllowed: cpu: "2000m" memory: "2048Mi" 4、查看部署结果 [root@k8s-node001 examples]# kubectl get vpa -n vpa NAME AGE nginx-vpa 2m34s 5、使用describe查看vpa详情,主要关注Container Recommendations [root@k8s-node001 examples]# kubectl describe vpa nginx-vpa -n vpa Name: nginx-vpa Namespace: vpa ....略去10000字 哈哈...... Update Policy: Update Mode: Off Status: Conditions: Last Transition Time: 2020-09-28T04:04:25Z Status: True Type: RecommendationProvided Recommendation: Container Recommendations: Container Name: nginx Lower Bound: Cpu: 250m Memory: 262144k Target: Cpu: 250m Memory: 262144k Uncapped Target: Cpu: 25m Memory: 262144k Upper Bound: Cpu: 803m Memory: 840190575 Events: <none>
Lower Bound: 下限值 Target: 推荐值 Upper Bound: 上限值 Uncapped Target: 如果没有为VPA提供最小或最大边界,则表示目标利用率 上述结果表明,推荐的 Pod 的 CPU 请求为 25m,推荐的内存请求为 262144k 字节。
# ab -c 100 -n 10000000 http://192.168.127.124:32621/ This is ApacheBench, Version 2.3 <$Revision: 1843412 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 192.168.127.124 (be patient) Completed 1000000 requests Completed 2000000 requests Completed 3000000 requests
# kubectl describe vpa nginx-vpa -n vpa |tail -n 20 Conditions: Last Transition Time: 2021-06-28T04:04:25Z Status: True Type: RecommendationProvided Recommendation: Container Recommendations: Container Name: nginx Lower Bound: Cpu: 250m Memory: 262144k Target: Cpu: 476m Memory: 262144k Uncapped Target: Cpu: 476m Memory: 262144k Upper Bound: Cpu: 2 Memory: 387578728 Events: <none>
# kubectl apply -f nginx-vpa.yaml deployment.apps/nginx created # cat nginx-vpa.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx namespace: vpa spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx resources: requests: cpu: 100m memory: 50Mi # kubectl get po -n vpa NAME READY STATUS RESTARTS AGE nginx-7ff65f974c-f4vgl 1/1 Running 0 114s nginx-7ff65f974c-v9ccx 1/1 Running 0 114s
updateMode: "Auto"
和name: nginx-vpa-2
# cat nginx-vpa-demo.yaml apiVersion: autoscaling.k8s.io/v1beta2 kind: VerticalPodAutoscaler metadata: name: nginx-vpa-2 namespace: vpa spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: nginx updatePolicy: updateMode: "Auto" resourcePolicy: containerPolicies: - containerName: "nginx" minAllowed: cpu: "250m" memory: "100Mi" maxAllowed: cpu: "2000m" memory: "2048Mi" # kubectl apply -f nginx-vpa-demo.yaml verticalpodautoscaler.autoscaling.k8s.io/nginx-vpa created # kubectl get vpa -n vpa NAME AGE nginx-vpa-2 9s
# ab -c 1000 -n 100000000 http://192.168.127.124:32621/
# kubectl describe vpa nginx-vpa-2 -n vpa |tail -n 30 Min Allowed: Cpu: 250m Memory: 100Mi Target Ref: API Version: apps/v1 Kind: Deployment Name: nginx Update Policy: Update Mode: Auto Status: Conditions: Last Transition Time: 2021-06-28T04:48:25Z Status: True Type: RecommendationProvided Recommendation: Container Recommendations: Container Name: nginx Lower Bound: Cpu: 250m Memory: 262144k Target: Cpu: 476m Memory: 262144k Uncapped Target: Cpu: 476m Memory: 262144k Upper Bound: Cpu: 2 Memory: 262144k Events: <none>
Target变成了Cpu: 587m ,Memory: 262144k
5、查看event事件
~]# kubectl get event -n vpa LAST SEEN TYPE REASON OBJECT MESSAGE 33m Normal Pulling pod/nginx-7ff65f974c-f4vgl Pulling image "nginx" 33m Normal Pulled pod/nginx-7ff65f974c-f4vgl Successfully pulled image "nginx" in 15.880996269s 33m Normal Created pod/nginx-7ff65f974c-f4vgl Created container nginx 33m Normal Started pod/nginx-7ff65f974c-f4vgl Started container nginx 26m Normal EvictedByVPA pod/nginx-7ff65f974c-f4vgl Pod was evicted by VPA Updater to apply resource recommendation. 26m Normal Killing pod/nginx-7ff65f974c-f4vgl Stopping container nginx 35m Normal Scheduled pod/nginx-7ff65f974c-hnzr5 Successfully assigned vpa/nginx-7ff65f974c-hnzr5 to k8s-node005 35m Normal Pulling pod/nginx-7ff65f974c-hnzr5 Pulling image "nginx" 34m Normal Pulled pod/nginx-7ff65f974c-hnzr5 Successfully pulled image "nginx" in 40.750855715s 34m Normal Scheduled pod/nginx-7ff65f974c-v9ccx Successfully assigned vpa/nginx-7ff65f974c-v9ccx to k8s-node004 33m Normal Pulling pod/nginx-7ff65f974c-v9ccx Pulling image "nginx" 33m Normal Pulled pod/nginx-7ff65f974c-v9ccx Successfully pulled image "nginx" in 15.495315629s 33m Normal Created pod/nginx-7ff65f974c-v9ccx Created container nginx 33m Normal Started pod/nginx-7ff65f974c-v9ccx Started container nginx
~]# kubectl describe po nginx-7ff65f974c-2m9zl -n vpa Name: nginx-7ff65f974c-2m9zl Namespace: vpa Priority: 0 Node: k8s-node004/192.168.100.184 Start Time: June, 28 Sep 2021 00:46:19 -0400 Labels: app=nginx pod-template-hash=7ff65f974c Annotations: cni.projectcalico.org/podIP: 100.67.191.53/32 vpaObservedContainers: nginx vpaUpdates: Pod resources updated by nginx-vpa: container 0: cpu request, memory request Status: Running IP: 100.67.191.53 IPs: IP: 100.67.191.53 Controlled By: ReplicaSet/nginx-7ff65f974c Containers: nginx: Container ID: docker://c96bcd07f35409d47232a0bf862a76a56352bd84ef10a95de8b2e3f6681df43d Image: nginx Image ID: docker-pullable://nginx@sha256:c628b67d21744fce822d22fdcc0389f6bd763daac23a6b77147d0712ea7102d0 Port: <none> Host Port: <none> State: Running Started: June, 28 Sep 2021 00:46:38 -0400 Ready: True Restart Count: 0 Requests: cpu: 476m memory: 262144k
requests: cpu: 100m memory: 50Mi
CA项目托管地址 :https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
节点的初始化: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/
参考博文 :https://mp.weixin.qq.com/s/GKS3DJHm4p0Tjtj8nJRGmA
HPA delay: 1m30s + CA delay: 0m30s + Cloud provider: 4m + Container runtime: 0m30s + ========================= Total 6m30s
(即使调小了上述设置,依然会受云服务商的时间限制)
HPA 的刷新时间,默认 15 秒,通过 --horizontal-pod-autoscaler-sync-period 标志控制; Metrics Server 的指标抓取时间,默认 60 秒,通过 metric-resolution 控 CA 的扫描间隔,默认 10 秒,通过 scan-interval 控制; 节点上缓存镜像,比如 kube-fledged等工具;