k8s之多方面多维度的资源隔离和限制(namespace,LimitRange,ResourceQuota)
k8s之多方面多维度的资源隔离和限制(namespace,LimitRange,ResourceQuota)
首先我们能想到的资源隔离就是namespace，这不知道是不是大家的第一反应，反正我是的，，哈哈哈
namespace的神奇之处
大家可以想象一下，把每个项目组拥有着不同的命名空间，操作着属于自己组内的资源对象，是不是非常的方便管理呢，好的，大概就是这样哈，简单理解，接下来我们进行一下简单的测试，来验证下我们的猜想，let,s go
测试下nmespace的资源隔离
创建一个新的namespace
[root@k8s-master1 ~]# kubectl create namespace dev
namespace/dev created
[root@k8s-master1 ~]# kubectl get namespace -A
NAME STATUS AGE
default Active 4d13h
dev Active 2m51s
kube-node-lease Active 4d13h
kube-public Active 4d13h
kube-system Active 4d13h
kubernetes-dashboard Active 4d11h
在当前的namespace里面创建两个不同的pod应用
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor-login
[root@k8s-master1 ~]# cat deployment-tomcat.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: tomcat
name: tomcat
spec:
replicas: 4
selector:
matchLabels:
app: tomcat
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: tomcat
spec:
containers:
- image: 10.245.4.88:8888/base-images/tomcat
name: tomcat
resources: {}
imagePullSecrets:
- name: harbor-login
status: {}
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
deployment.apps/nginx created
[root@k8s-master1 ~]# kubectl apply -f deployment-tomcat.yaml
deployment.apps/tomcat created
[root@k8s-master1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-5896cbffc6-4vcdv 1/1 Running 0 11m 10.244.169.159 k8s-node2 <none> <none>
nginx-5896cbffc6-jfzsz 1/1 Running 0 11m 10.244.159.163 k8s-master1 <none> <none>
nginx-5896cbffc6-n7n2x 1/1 Running 0 11m 10.244.36.90 k8s-node1 <none> <none>
nginx-5896cbffc6-rt9h7 1/1 Running 0 11m 10.244.36.91 k8s-node1 <none> <none>
tomcat-596db6d496-l4dvs 1/1 Running 0 11m 10.244.169.160 k8s-node2 <none> <none>
tomcat-596db6d496-nhc4c 1/1 Running 0 11m 10.244.159.164 k8s-master1 <none> <none>
tomcat-596db6d496-qfkwr 1/1 Running 0 11m 10.244.36.92 k8s-node1 <none> <none>
tomcat-596db6d496-zd8dp 1/1 Running 0 8m27s 10.244.36.94 k8s-node1 <none> <none>
测试同一个命名空间下pod与pod是否可以互通
其实这个不用测试的，一般情况下都是互通的，但是为了效果，我们还是要进行一下测试，来验证下我们资源的隔离
进入一个Pod里面,以名称和ip地址来进行互相url访问
[root@k8s-master1 ~]# kubectl exec -it tomcat-596db6d496-zd8dp /bin/bash ###进入到tomcat容器里面
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@tomcat-596db6d496-zd8dp:/usr/local/tomcat# curl nginx ##curl访问名称为nginx的deployment，发现可以访问到
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@tomcat-596db6d496-zd8dp:/usr/local/tomcat# curl 10.244.169.159 ###使用ip地址进行访问，同样也是可以访问通的
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@tomcat-596db6d496-zd8dp:/usr/local/tomcat# exit
exit
- 结论：同命名空间下的pod是可以通过名称来相互访问的
测试不同namespace下pod与pod是否互通
在dev命名空间下创建一个pod
###h还是使用上面的deploymen-nginx.yaml文件，发现是可以创建成功的，说明不同的namescpace下可以创建相同的名称deployment
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml -n dev
deployment.apps/nginx created
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-5896cbffc6-52rql 1/1 Running 0 8s
nginx-5896cbffc6-8znq5 1/1 Running 0 8s
nginx-5896cbffc6-fvdbf 1/1 Running 0 8s
nginx-5896cbffc6-jrppp 1/1 Running 0 8s
[root@k8s-master1 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-5896cbffc6-4vcdv 1/1 Running 0 22m
default nginx-5896cbffc6-jfzsz 1/1 Running 0 22m
default nginx-5896cbffc6-n7n2x 1/1 Running 0 22m
default nginx-5896cbffc6-rt9h7 1/1 Running 0 22m
default tomcat-596db6d496-l4dvs 1/1 Running 0 22m
default tomcat-596db6d496-nhc4c 1/1 Running 0 22m
default tomcat-596db6d496-qfkwr 1/1 Running 0 22m
default tomcat-596db6d496-zd8dp 1/1 Running 0 19m
dev nginx-5896cbffc6-52rql 1/1 Running 0 27s
dev nginx-5896cbffc6-8znq5 1/1 Running 0 27s
dev nginx-5896cbffc6-fvdbf 1/1 Running 0 27s
dev nginx-5896cbffc6-jrppp 1/1 Running 0 27s
kube-system calico-kube-controllers-8db96c76-w4njz 1/1 Running 1 (21h ago) 3d13h
kube-system calico-node-lxfbc 1/1 Running 1 (21h ago) 3d13h
kube-system calico-node-v2fbv 1/1 Running 1 (21h ago) 3d13h
kube-system calico-node-vd6v2 1/1 Running 19 (21h ago) 3d13h
kube-system calico-node-xvqmt 1/1 Running 1 (21h ago) 3d13h
kube-system coredns-7b9f9f5dfd-6fhwh 1/1 Running 2 (21h ago) 4d11h
kubernetes-dashboard dashboard-metrics-scraper-5594697f48-kw9zt 1/1 Running 3 (21h ago) 4d12h
kubernetes-dashboard kubernetes-dashboard-686cc7c688-cn92d 1/1 Running 2 (21h ago) 38h
删除default命名空间下的nginx的deployment
为为了避免一会curl访问的时候不知道访问的是那个命名空间下 的
[root@k8s-master1 ~]# kubectl delete -f deployment-nginx.yaml
deployment.apps "nginx" deleted
[root@k8s-master1 ~]# kubectl get pod -n default
NAME READY STATUS RESTARTS AGE
tomcat-596db6d496-l4dvs 1/1 Running 0 27m
tomcat-596db6d496-nhc4c 1/1 Running 0 27m
tomcat-596db6d496-qfkwr 1/1 Running 0 27m
tomcat-596db6d496-zd8dp 1/1 Running 0 24m
在default命名空间下curl访问dev命名空间下资源
[root@k8s-master1 ~]# kubectl get pod -n dev -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-5896cbffc6-52rql 1/1 Running 0 3m11s 10.244.224.24 k8s-master2 <none> <none>
nginx-5896cbffc6-8znq5 1/1 Running 0 3m11s 10.244.169.163 k8s-node2 <none> <none>
nginx-5896cbffc6-fvdbf 1/1 Running 0 3m11s 10.244.159.165 k8s-master1 <none> <none>
nginx-5896cbffc6-jrppp 1/1 Running 0 3m11s 10.244.224.23 k8s-master2 <none> <none>
[root@k8s-master1 ~]# kubectl get pod -n default
NAME READY STATUS RESTARTS AGE
tomcat-596db6d496-l4dvs 1/1 Running 0 27m
tomcat-596db6d496-nhc4c 1/1 Running 0 27m
tomcat-596db6d496-qfkwr 1/1 Running 0 27m
tomcat-596db6d496-zd8dp 1/1 Running 0 24m
[root@k8s-master1 ~]# kubectl exec -it tomcat-596db6d496-l4dvs /bin/bash ###进入default命名空间下面的容器里面
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@tomcat-596db6d496-l4dvs:/usr/local/tomcat# curl nginx ###访问dev命名空间下的Nginx，发现是访问不了的
^C
root@tomcat-596db6d496-l4dvs:/usr/local/tomcat# curl 10.244.224.24 ###但是我们使用Ip发现是可以正常访问成功的
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
- 结论1：不同的namescpace下面可以存在名字一样的资源
- 结论2：不同的namespace下面名字是相对隔离的，但是Ip地址还是通着的
- 结论3：namespace只是对名字进行了隔离，但是ip地址没有，这样也增加了更多大灵活性
对pod进行相应的资源限制和管理
- Requests:期望被分配到的，可以正常运行的资源大小，调度器会根据requests值来参与调度计算来进行调度
- Limits:限制资源最大上限，同时如果Node节点上面发生资源争，kubelet也会根据这个值来进行优化调整
### 清理调之前测试验证使用的容器
[root@k8s-master1 ~]# kubectl delete -f deployment-nginx.yaml -n dev
deployment.apps "nginx" deleted
[root@k8s-master1 ~]# kubectl delete -f deployment-tomcat.yaml
deployment.apps "tomcat" deleted
[root@k8s-master1 ~]# kubectl get pod
No resources found in default namespace.
测试pod的资源限制效果
编写yaml文件，添加限制参数
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 100Mi
cpu: 200m
limits:
memory: 200Mi
cpu: 500m
imagePullSecrets:
- name: harbor-login
查看当前机器剩余的资源情况
###比如查看Node1节点的使用情况
[root@k8s-master1 ~]# kubectl describe node k8s-node1
Name: k8s-node1
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
***********省略*******************************
kubelet is posting ready status
Addresses:
InternalIP: 10.245.4.3
Hostname: k8s-node1
Capacity: ####资源的容量
cpu: 2 ###cpu大小
ephemeral-storage: 17394Mi ###磁盘大小
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 1865308Ki ###内存大小
pods: 110 ###可以容纳pod容量，注意呀这可不是说能创建110个pod的，这个也得根据实际情况来说的，但是最大不能超过这个
Allocatable: #####可分配的资源
cpu: 2 ###可分配cpu的大小
ephemeral-storage: 16415037823 ###可分配磁盘的大小
hugepages-1Gi: 0 ####大页内存
hugepages-2Mi: 0 ####大页内存
memory: 1762908Ki ##可分配内存的大小
pods: 110
System Info:
Machine ID: 5b2e42d65e1c4fd99f8c541538c1c268
System UUID: B3BA4D56-BECB-1461-0470-BFF8A761EEDA
Boot ID: 1da3d5ad-e770-4c08-a029-a94703742702
Kernel Version: 3.10.0-862.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.9
Kubelet Version: v1.22.4
Kube-Proxy Version: v1.22.4
PodCIDR: 10.244.2.0/24
PodCIDRs: 10.244.2.0/24
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system calico-node-f7gqq 250m (12%) 0 (0%) 0 (0%) 0 (0%) 10m ###可以查看具体pod占用的资源
kube-system coredns-7b9f9f5dfd-9f8lm 100m (5%) 0 (0%) 70Mi (4%) 512Mi (29%) 10m ###可以查看具体pod占用的资源
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 350m (17%) 0 (0%) ####cpu已经使用了百分之十七
memory 70Mi (4%) 512Mi (29%) ####内存已经占用了百分之四，还有一个pod里面有linit限制为512Mi
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
[root@k8s-master1 ~]#
创建资源限制pod应用
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
deployment.apps/nginx created
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-66c78dc668-cvjtk 1/1 Running 0 58s
nginx-66c78dc668-lltrd 1/1 Running 0 58s
nginx-66c78dc668-n2nrf 1/1 Running 0 58s
nginx-66c78dc668-qjn5b 1/1 Running 0 58s
####我们更改下资源限制，看看Pod会发生什么
####将limits内存改成500M
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 100Mi
cpu: 200m
limits:
memory: 500Mi
cpu: 500m
imagePullSecrets:
- name: harbor-login
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-9b46cf8dd-475vw 1/1 Running 0 4s ###看age，可以发现pod已经重启了，那么为什么呢
nginx-9b46cf8dd-gqp9q 1/1 Running 0 2s
nginx-9b46cf8dd-qdrg7 1/1 Running 0 2s
nginx-9b46cf8dd-tc2dg 1/1 Running 0 4s
搞清楚为什么资源配置pod都发生了什么
###先看看pod的都在那个节点上
[root@k8s-master1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9b46cf8dd-475vw 1/1 Running 0 2m35s 10.244.159.170 k8s-master1 <none> <none> ##以他为列
nginx-9b46cf8dd-gqp9q 1/1 Running 0 2m33s 10.244.224.28 k8s-master2 <none> <none>
nginx-9b46cf8dd-qdrg7 1/1 Running 0 2m33s 10.244.169.168 k8s-node2 <none> <none>
nginx-9b46cf8dd-tc2dg 1/1 Running 0 2m35s 10.244.36.100 k8s-node1 <none> <none>
##可以看到nginx-9b46cf8dd-475vw是运行的master1上面的
####到master1这个Node上去
###其实我们的容器也是docker来进行启动的，当我们修改了Pod相关的资源配置后，docker需要重新启动一个容器来完成资源的隔离，这也就是pod为什么会重启的原因
[root@k8s-master1 ~]# docker ps -a | grep nginx
7b53d10a8e27 605c77e624dd "/docker-entrypoint.…" 5 minutes ago Up 5 minutes k8s_nginx_nginx-9b46cf8dd-475vw_default_240f3c7f-4a9b-441c-9fd7-49bbc9b9d960_0
*******************太多省略*****************************
[root@k8s-master1 ~]# docker inspect 7b53d10a8e27
[
{
"Id": "7b53d10a8e27d2c78312692fafb9afd22717630148e45edfc36b9041521f890f",
"Created": "2022-12-22T13:26:59.756903714Z",
"Path": "/docker-entrypoint.sh",
********************省略*********************************************************************
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 204, #######cpu共享，因为咱们的yaml文件里面配置的是200m,也就是200/1000*1024=204.8
###CpuShares,这个值是一个权重值，当资源发生争抢时，docker会按照这个值和其他的容器的这个值的比例来合理调整可用资源的大小，
####比如别人是2，这个是1，2:1的比例，当还有6个资源可用的时候，6*1/3=2个，这个就可以分到2个
"Memory": 524288000, ###524288000/1024/1024=500M #为我们limits的内存
"NanoCpus": 0,
"CgroupParent": "/kubepods/burstable/pod240f3c7f-4a9b-441c-9fd7-49bbc9b9d960",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 100000, ####docker的默认值，和下面的CpuQuota一起使用，代表着100毫秒最多分配给容器500m
"CpuQuota": 50000, ####是我们在linits做的设置，50000/1000=500m
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
***************************省略***************************************
测试容器内存达到limits上限会发生什么
还是在上面的容器上，我们模拟下内存达到上限
[root@k8s-master1 ~]# docker exec -it 7b53d10a8e27 /bin/bash
root@nginx-9b46cf8dd-475vw:/# cat >> test.sh << EOF
> a="dadasdfsafdsafdsfsdafsdf"
> while 2>1
> do
> a="\$a\$a"
> sleep 0.1
> done
> EOF
root@nginx-9b46cf8dd-475vw:/# cat test.sh
a="dadasdfsafdsafdsfsdafsdf"
while 2>1
do
a="$a$a"
sleep 0.1
done
root@nginx-9b46cf8dd-475vw:/# sh test.sh
Killed ###可以发现我们的进程会被主动killd
[root@k8s-master1 ~]# docker ps -a | grep 7b53d10a8e27
####但是我们的容器还在没有重启,说明会kill掉占用内存最高的那个进程
7b53d10a8e27 605c77e624dd "/docker-entrypoint.…" 42 minutes ago Up 42 minutes k8s_nginx_nginx-9b46cf8dd-475vw_default_240f3c7f-4a9b-441c-9fd7-49bbc9b9d960_0
- 结论：当发现有占用内存较高的进行,k8s会主动的杀掉这个进程
测试容器将cpu达到limit上限会发生什么
###调整部署应用
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 1000Mi
cpu: 200m
limits:
memory: 5000Mi
cpu: 500m ###cpu限制到了500m
imagePullSecrets:
- name: harbor-login
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-556585848b-4ds7x 1/1 Running 0 9m27s 10.244.159.173 k8s-master1 <none> <none>
nginx-556585848b-8b7dt 1/1 Running 0 9m26s 10.244.169.170 k8s-node2 <none> <none>
nginx-556585848b-gszt5 1/1 Running 0 9m25s 10.244.224.30 k8s-master2 <none> <none>
nginx-556585848b-ksbds 1/1 Running 0 9m27s 10.244.36.102 k8s-node1 <none> <none>
[root@k8s-master1 ~]# docker ps -a | grep nginx ###在master节点找到相对应得容器
d153182a5410 605c77e624dd "/docker-entrypoint.…" 9 minutes ago Up 9 minutes k8s_nginx_nginx-556585848b-4ds7x_default_ee2915cc-b347-4187-b1b9-4e3f1ddd83ba_0
*****************省略******************
####进到容器里面模拟占用cpu
[root@k8s-master1 ~]# docker exec -it d153182a5410 /bin/bash
root@nginx-556585848b-4ds7x:/# dd if=/dev/zero of=/dev/null & ####这玩意最耗cpu，我们夸张一点干8个进程
[1] 38
root@nginx-556585848b-4ds7x:/# dd if=/dev/zero of=/dev/null &
[2] 39
root@nginx-556585848b-4ds7x:/# dd if=/dev/zero of=/dev/null &
[3] 40
root@nginx-556585848b-4ds7x:/# dd if=/dev/zero of=/dev/null &
[4] 41
root@nginx-556585848b-4ds7x:/# dd if=/dev/zero of=/dev/null &
[5] 42
root@nginx-556585848b-4ds7x:/# dd if=/dev/zero of=/dev/null &
[6] 43
root@nginx-556585848b-4ds7x:/# dd if=/dev/zero of=/dev/null &
[7] 44
root@nginx-556585848b-4ds7x:/# dd if=/dev/zero of=/dev/null &
[8] 45
root@nginx-556585848b-4ds7x:/# exit
exit
[root@k8s-master1 ~]# docker stats d153182a5410
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d153182a5410 k8s_nginx_nginx-556585848b-4ds7x_default_ee2915cc-b347-4187-b1b9-4e3f1ddd83ba_0 51.70% 4.898MiB / 1.779GiB 0.27% 0B / 0B 0B / 0B 11
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d153182a5410 k8s_nginx_nginx-556585848b-4ds7x_default_ee2915cc-b347-4187-b1b9-4e3f1ddd83ba_0 52.30% 4.898MiB / 1.779GiB 0.27% 0B / 0B 0B / 0B 11
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d153182a5410 k8s_nginx_nginx-556585848b-4ds7x_default_ee2915cc-b347-4187-b1b9-4e3f1ddd83ba_0 52.30% 4.898MiB / 1.779GiB 0.27% 0B / 0B 0B / 0B 11
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d153182a5410 k8s_nginx_nginx-556585848b-4ds7x_default_ee2915cc-b347-4187-b1b9-4e3f1ddd83ba_0 50.98% 4.898MiB / 1.779GiB 0.27% 0B / 0B 0B / 0B 11
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d153182a5410 k8s_nginx_nginx-556585848b-4ds7x_default_ee2915cc-b347-4187-b1b9-4e3f1ddd83ba_0 50.98% 4.898MiB / 1.779GiB 0.27% 0B / 0B 0B / 0B 11
### 可以看到cpu一直稳定在50%,这是因为cpu是一种可压缩得资源是允许超卖得，但是他也不会超出limits上限值，进程也不会被Kill掉
- 结论：当一个进程把cpu撑起来的话，不会被killd掉，会稳定在limits限制，因为cpu可以压缩和超卖
测试将linits值调整非常高，看下效果
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 100Mi
cpu: 200m
limits:
memory: 500Mi
cpu: 500 ###不带m的话代表着是500个核心
imagePullSecrets:
- name: harbor-login
###重新应用一下
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-5556f6dc97-dn4hf 1/1 Running 0 4s
nginx-5556f6dc97-lqfqt 1/1 Running 0 4s
nginx-5556f6dc97-npvf8 0/1 ContainerCreating 0 2s
nginx-5556f6dc97-trz8b 0/1 ContainerCreating 0 1s
nginx-9b46cf8dd-qdrg7 1/1 Running 0 46m
###可以发现已经被调度起来了
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-5556f6dc97-dn4hf 1/1 Running 0 9s
nginx-5556f6dc97-lqfqt 1/1 Running 0 9s
nginx-5556f6dc97-npvf8 1/1 Running 0 7s
nginx-5556f6dc97-trz8b 1/1 Running 0 6s
- 结论： 调度策略是不参照linits值得
测试将内存requests值调大，看下效果
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 100Gi ####requests为100G
cpu: 200m
limits:
memory: 500Mi
cpu: 500
imagePullSecrets:
- name: harbor-login
####重新应用一下
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
The Deployment "nginx" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: "100Gi": must be less than or equal to memory limit
###意外收获，request得值不能大于limit值，否则不会参与调度，连api-server都过不去
###继续调整资源配置
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 100Gi ####request 100G
cpu: 200m
limits:
memory: 500Gi ####linits 500G
cpu: 500
imagePullSecrets:
- name: harbor-login
####重新应用一下
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-5556f6dc97-dn4hf 1/1 Running 0 7m45s
nginx-5556f6dc97-lqfqt 1/1 Running 0 7m45s
nginx-5556f6dc97-trz8b 1/1 Running 0 7m42s
nginx-5999cd5549-2qtwp 0/1 Pending 0 6s ###pending状态
nginx-5999cd5549-7nbkx 0/1 Pending 0 6s ###pending状态
因为我的pod是4个副本，按照K8s滚动更新得策略，更新前两个如果卡住，就不会更新后面得
###看下为什么滚动更新不了
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-5556f6dc97-dn4hf 1/1 Running 0 9m13s
nginx-5556f6dc97-lqfqt 1/1 Running 0 9m13s
nginx-5556f6dc97-trz8b 1/1 Running 0 9m10s
nginx-5999cd5549-2qtwp 0/1 Pending 0 94s
nginx-5999cd5549-7nbkx 0/1 Pending 0 94s
[root@k8s-master1 ~]# kubectl describe pod nginx-5999cd5549-7nbkx | tail -10
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 28s (x6 over 4m30s) default-scheduler 0/4 nodes are available: 4 Insufficient memory.
###4个内存不足，小编是4个节点得集群
- 结论1：当内存的requests设置较大，是无法进行调度的
- 结论2：requests值必须小于limit值，否则也会无法调度
测试将cpu得requests值调大，看下效果
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 100Mi
cpu: 200 ###调整成200核
limits:
memory: 500Mi
cpu: 500
imagePullSecrets:
- name: harbor-login
###重新应用一下
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-5556f6dc97-dn4hf 1/1 Running 0 17m
nginx-5556f6dc97-lqfqt 1/1 Running 0 17m
nginx-5556f6dc97-trz8b 1/1 Running 0 17m
nginx-6b895b54fb-d5n42 0/1 Pending 0 6s
nginx-6b895b54fb-qhjhn 0/1 Pending 0 6s
[root@k8s-master1 ~]# kubectl describe pod nginx-6b895b54fb-qhjhn | tail -10
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 31s default-scheduler 0/4 nodes are available: 4 Insufficient cpu.
###4个cpu不足，小编是4个节点得集群
- 结论： cpu的requests设置较大时，无法满足调度条件，也是不会被调度的
测试下极个别pod资源不够用得情况
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 5
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 1000Mi
###每个pod得1000m,运行5个副本，小编4台机器，每台是2G内存，加上自身消耗，5个pod应该不会全部运行
cpu: 200m
limits:
memory: 5000Mi
cpu: 500m
imagePullSecrets:
- name: harbor-login
###重新应用一下
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-5556f6dc97-trz8b 1/1 Running 0 25m
nginx-556585848b-4ds7x 1/1 Running 0 5s
nginx-556585848b-8b7dt 1/1 Running 0 4s
nginx-556585848b-cd7ns 0/1 Pending 0 2s
nginx-556585848b-gszt5 0/1 ContainerCreating 0 3s
nginx-556585848b-ksbds 1/1 Running 0 5s
[root@k8s-master1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-556585848b-4ds7x 1/1 Running 0 10s
nginx-556585848b-8b7dt 1/1 Running 0 9s
nginx-556585848b-cd7ns 0/1 Pending 0 7s ###可以看到一个pod是pending状态
nginx-556585848b-gszt5 1/1 Running 0 8s
nginx-556585848b-ksbds 1/1 Running 0 10s
[root@k8s-master1 ~]# kubectl describe pod nginx-556585848b-cd7ns | tail -10
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 68s (x3 over 71s) default-scheduler 0/4 nodes are available: 4 Insufficient memory.
###4个内存不足
- 结论：requests，就是调度得依据，那怕你机器上面可以运行一个pod，但是不满足我得requests要求，依然是不能够被调度得
资源分配得优先级等级划分
- requests 等于 limits:说明该应用是可靠得
- 如果不设置这些参数或者不设置其中一个，那就是不可靠得（不建议，当资源不足得时候，优先会杀掉这些服务
- requests < limit：说明该应用是比较可靠得
几点考虑
1、上面我们所学的requests和limits，都是用户自己设置得
2、如果有人将requests设置得超出了我们得资源大小，是不是这些pod会永远也调度不起来，很不好得客户体验
3、如果说requests设置得合理，但是linit设置的超大，突然客户程序出现了大量占用资源情况，又没达到limits上限，是不是也很浪费我们得资源呢
上面得情况不要怕，k8s早就帮忙想好了，接着往下看就知道了
利用LinitsRange来对资源配置进行限制
limitsRange是对命名空间所言得，所以我们还是要用到上面得dev命名空间
[root@k8s-master1 ~]# vi limitrange.yaml
[root@k8s-master1 ~]# cat limitrange.yaml
apiVersion: v1
kind: LimitRange ###资源类型
metadata:
name: test-limitrange
spec:
limits:
- max: ###该命名空间下创建资源得时候最大得限制
cpu: 600m
memory: 1200Mi
min: ###该命名空间下创建资源得时候最小得资源大小
cpu: 100m
memory: 100Mi
maxLimitRequestRatio: ####创建资源得时候，limits和requests得最大比值
cpu: 3
memory: 2
type: Pod #####以上是针对pod类型资源得，下面得是是针对容器资源得，因为一个pod里面会有多个容器得现象，所以他是没有默认得，但是容器有
- default: ###创建容器得时候默认得最大限制
cpu: 200m
memory: 200Mi
defaultRequest: ###创建容器得时候默认得最小资源配额
cpu: 200m
memory: 100Mi
max: ####创建得容器得时候最大允许得配置大小
cpu: 200m
memory: 1000Mi
min: ###创建容器得时候最小得允许得配额大小
cpu: 100m
memory: 100Mi
maxLimitRequestRatio: ####创建容器得时候,limit和requests得最大比值
cpu: 2
memory: 4
type: Container
###指定命名空间进行创建limitrange。
[root@k8s-master1 ~]# kubectl apply -f limitrange.yaml -n dev
limitrange/test-limitrange created
###查看dev命名空间下面得所有limits，可以在这里看到我们所有得资源限制配置
[root@k8s-master1 ~]# kubectl describe limits -n dev
Name: test-limitrange
Namespace: dev
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Pod memory 100Mi 1200Mi - - 2
Pod cpu 100m 600m - - 3
Container cpu 100m 200m 200m 200m 2
Container memory 100Mi 1000Mi 100Mi 200Mi 4
测试不加requests和limits创建资源，看下效果
####注释掉我们得资源限制
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
# resources:
# requests:
# memory: 1000Mi
# cpu: 200m
# limits:
# memory: 5000Mi
# cpu: 500m
imagePullSecrets:
- name: harbor-login
###部署到dev空间下
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml -n dev
deployment.apps/nginx created
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-5896cbffc6-5t9z8 1/1 Running 0 6s
nginx-5896cbffc6-dlmgd 1/1 Running 0 6s
nginx-5896cbffc6-mt4jn 1/1 Running 0 6s
nginx-5896cbffc6-tpsnq 1/1 Running 0 6s
###查看当前pod资源配置
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-5896cbffc6-5t9z8 1/1 Running 0 3m33s
nginx-5896cbffc6-dlmgd 1/1 Running 0 3m33s
nginx-5896cbffc6-mt4jn 1/1 Running 0 3m33s
nginx-5896cbffc6-tpsnq 1/1 Running 0 3m33s
[root@k8s-master1 ~]# kubectl get pod -n dev -o yaml | grep -A 15 "spec"
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
imagePullPolicy: IfNotPresent
name: nginx
resources:
limits: ####可以看到已经默认都添加了资源得限制
cpu: 200m
memory: 200Mi
requests:
cpu: 200m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
--
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
imagePullPolicy: IfNotPresent
name: nginx
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 200m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
--
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
imagePullPolicy: IfNotPresent
name: nginx
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 200m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
--
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
imagePullPolicy: IfNotPresent
name: nginx
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 200m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
- 结论：如果配置了linirrange,在创建资源没有添加资源限制，那么默认会按照limitrange里面定义得最大和最小来进行限制
测试没有按照比例分配创建资源，看下效果
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 200Mi ###当前比例为3，超出了当前规定得2
cpu: 200m
limits:
memory: 600Mi
cpu: 500m
imagePullSecrets:
- name: harbor-login
###重新部署下应用
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-5896cbffc6-5t9z8 1/1 Running 0 12m
nginx-5896cbffc6-mt4jn 1/1 Running 0 12m
nginx-5896cbffc6-tpsnq 1/1 Running 0 12m
[root@k8s-master1 ~]# kubectl get deployment -n dev
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/4 0 3 12m ####UP-TO-DATE为0，说明deployment没有更新
[root@k8s-master1 ~]# kubectl get deployment -n dev -o yaml | grep message ####看看deployment得相关信息
message: Deployment has minimum availability.
message: 'pods "nginx-54765fd77c-lct5r" is forbidden: [memory max limit to request
message: ReplicaSet "nginx-54765fd77c" is progressing.
[root@k8s-master1 ~]# kubectl get deployment -n dev -o yaml | grep -A 6 message
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2022-12-22T15:44:46Z"
lastUpdateTime: "2022-12-22T15:44:46Z"
message: 'pods "nginx-54765fd77c-lct5r" is forbidden: [memory max limit to request
ratio per Pod is 2, but provided ratio is 3.000000, maximum cpu usage per ####这边有提示了，比例为2，当前为3
Container is 200m, but limit is 500m, cpu max limit to request ratio per Container
is 2, but provided ratio is 2.500000]'
reason: FailedCreate ###所以失败
status: "True"
type: ReplicaFailure
--
message: ReplicaSet "nginx-54765fd77c" is progressing.
reason: ReplicaSetUpdated
status: "True"
type: Progressing
observedGeneration: 2
readyReplicas: 3
replicas: 3
- 结论：当不符合limitrange相关比例配置得时候，是无法创建deployment得
测试超出设置得最大限额创建资源，看下效果
[root@k8s-master1 ~]# kubectl describe limits -n dev
Name: test-limitrange
Namespace: dev
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Pod memory 100Mi 1200Mi - - 2
Pod cpu 100m 600m - - 3
Container cpu 100m 200m 200m 200m 2
Container memory 100Mi 1000Mi 100Mi 200Mi 4
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 1800Mi ###limitrange上限是1200Mi
cpu: 200m
limits:
memory: 6000Mi
cpu: 500m
imagePullSecrets:
- name: harbor-login
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml -n dev
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-5896cbffc6-5t9z8 1/1 Running 0 19m
nginx-5896cbffc6-mt4jn 1/1 Running 0 19m
nginx-5896cbffc6-tpsnq 1/1 Running 0 19m
[root@k8s-master1 ~]# kubectl get deployment -n dev
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/4 0 3 20m
[root@k8s-master1 ~]# kubectl get deployment -n dev -o yaml | grep -A 6 "message"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2022-12-22T15:44:46Z"
lastUpdateTime: "2022-12-22T15:44:46Z"
message: 'pods "nginx-54765fd77c-lct5r" is forbidden: [memory max limit to request
ratio per Pod is 2, but provided ratio is 3.000000, maximum cpu usage per ####报比例错误，可以说明，比例是优先得
Container is 200m, but limit is 500m, cpu max limit to request ratio per Container
is 2, but provided ratio is 2.500000]'
reason: FailedCreate
status: "True"
type: ReplicaFailure
--
message: Created new replica set "nginx-9d7f8f5cb"
reason: NewReplicaSetCreated
status: "True"
type: Progressing
observedGeneration: 3
readyReplicas: 3
replicas: 3
####我们再改下资源配置，符合比例得情况
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 1800Mi
cpu: 200m
limits:
memory: 2000Mi
cpu: 500m
imagePullSecrets:
- name: harbor-login
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml -n dev
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-5896cbffc6-5t9z8 1/1 Running 0 22m ###pod还是没有变化
nginx-5896cbffc6-mt4jn 1/1 Running 0 22m
nginx-5896cbffc6-tpsnq 1/1 Running 0 22m
[root@k8s-master1 ~]# kubectl get deployment -n dev
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/4 0 3 23m ###deployment也没有更新
[root@k8s-master1 ~]# kubectl get deployment -n dev -o yaml | grep -A 6 "message"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2022-12-22T15:34:25Z"
lastUpdateTime: "2022-12-22T15:54:14Z"
message: Created new replica set "nginx-9d7f8f5cb"
reason: NewReplicaSetCreated
status: "True"
type: Progressing
- lastTransitionTime: "2022-12-22T15:57:14Z"
lastUpdateTime: "2022-12-22T15:57:14Z"
message: 'pods "nginx-7599cdff87-qbcxz" is forbidden: [maximum memory usage
per Pod is 1200Mi, but limit is 2097152k, maximum memory usage per Container
is 1000Mi, but limit is 2000Mi, maximum cpu usage per Container is 200m, but ###可以看到内存配置已经操作limitrange限制了
limit is 500m, cpu max limit to request ratio per Container is 2, but provided
ratio is 2.500000]'
reason: FailedCreate ###所以创建会失败
status: "True"
- 结论：如果创建资源得时候超出了不在限制之内得话，也是无法创建资源得。
- 结论2：优先会检查资源配置得比例，后对比资源限制得配置
再来几点考虑
1、如果咱们得团队过大，有多个命名空间，也就是不同的项目组使用不同的命令空间，有先对来说比较重要的，有相对来说不是那么重要的，那么怎么来限制他们呢
2、如果说相对来说不是那么重要的命名空间团队，创建了好多pod，没有了还在那边放着，还在继续创建资源怎么办，这岂不是又是一种浪费
当然了,k8s也想到了，继续向下面看
利用ResourceQuota来对命名空间下资源进行限额
大家可能有个困惑，就是ResourceQuota和limits都是来进行资源限制的，那该听谁的呢，其实不是ResourceQuota这个是一个总值，比如说pod数量，所有的pod数量加起来不能达到ResourceQuota限制。
[root@k8s-master1 ~]# vi ResourceQuota.yaml
[root@k8s-master1 ~]# cat ResourceQuota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: test-resourcequota
spec:
hard:
pods: 4 ###限制创建pod二点数量为4个
requests.cpu: 200m ###对cpu、内存进行限制
requests.memory: 400Mi
limits.cpu: 800m
limits.memory: 800Mi
##当然还有好多可以配置的，小编就不一一去验证了
[root@k8s-master1 ~]# kubectl apply -f ResourceQuota.yaml -n dev
resourcequota/test-resourcequota created
[root@k8s-master1 ~]# kubectl describe quota -n dev
Name: test-resourcequota
Namespace: dev
Resource Used Hard ####used是已经使用了的，hard是硬性要求的大小
-------- ---- ----
limits.cpu 0 800m
limits.memory 0 800Mi
pods 0 4
requests.cpu 0 200m
requests.memory 0 400Mi
测试下当资源配额量达到上限后，看看是什么效果
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 200Mi
cpu: 100m
limits:
memory: 400Mi
cpu: 100m
imagePullSecrets:
- name: harbor-login
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml -n dev
deployment.apps/nginx configured
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-68c84ffdc7-2jsx2 1/1 Running 0 2s
nginx-68c84ffdc7-76qlr 1/1 Running 0 4s
[root@k8s-master1 ~]# kubectl describe quota -n dev
Name: test-resourcequota
Namespace: dev
Resource Used Hard
-------- ---- ----
limits.cpu 200m 800m ##还未达到
limits.memory 800Mi 800Mi ###已经达到了上限
pods 2 4 ##当前已经运行了两个pod
requests.cpu 200m 200m ###已经达到了上限
requests.memory 400Mi 400Mi ###已经达到了上限
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx-1 ###更改下名字，确保不重复
name: nginx-1 ###更改下名字
spec:
replicas: 2
selector:
matchLabels:
app: nginx-1 ###更改下名字
template:
metadata:
creationTimestamp: null
labels:
app: nginx-1 ###更改下名字
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx-1 ###更改下名字
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 200Mi
cpu: 100m
limits:
memory: 400Mi
cpu: 100m
imagePullSecrets:
- name: harbor-login
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml -n dev
deployment.apps/nginx-1 created
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-68c84ffdc7-2jsx2 1/1 Running 0 3m51s
nginx-68c84ffdc7-76qlr 1/1 Running 0 3m53s
[root@k8s-master1 ~]# kubectl get deployment nginx-1 -n dev -o yaml | grep -A 6 "message"
message: Created new replica set "nginx-1-7dcf8cbc69"
reason: NewReplicaSetCreated
status: "True"
type: Progressing
- lastTransitionTime: "2022-12-22T16:44:50Z"
lastUpdateTime: "2022-12-22T16:44:50Z"
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: "False"
type: Available
- lastTransitionTime: "2022-12-22T16:44:50Z"
lastUpdateTime: "2022-12-22T16:44:50Z"
message: 'pods "nginx-1-7dcf8cbc69-cgxl7" is forbidden: exceeded quota: test-resourcequota, ###可以看到已经被限制了
requested: limits.memory=400Mi,requests.cpu=100m,requests.memory=200Mi, used:
limits.memory=800Mi,requests.cpu=200m,requests.memory=400Mi, limited: limits.memory=800Mi,requests.cpu=200m,requests.memory=400Mi'
reason: FailedCreate ### 所以失败
status: "True"
type: ReplicaFailure
observedGeneration: 1
- 结论：当一个资源达到配额限制的时候，也是不会被调度的
测试下当pod数量达到限额创建资源，看看效果
###清理当前的资源环境
[root@k8s-master1 ~]# kubectl delete -f deployment-nginx.yaml -n dev
[root@k8s-master1 ~]# kubectl delete deployment nginx -n dev
deployment.apps "nginx" deleted
[root@k8s-master1 ~]# kubectl get pod -n dev
No resources found in dev namespace.
##为了实验效果，更改下quota的yaml文件并重新应用
[root@k8s-master1 ~]# vi ResourceQuota.yaml
[root@k8s-master1 ~]# cat ResourceQuota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: test-resourcequota
spec:
hard:
pods: 4
requests.cpu: 2000m
requests.memory: 4000Mi
limits.cpu: 8000m
limits.memory: 8000Mi
[root@k8s-master1 ~]# kubectl apply -f ResourceQuota.yaml -n dev
resourcequota/test-resourcequota configured
[root@k8s-master1 ~]# kubectl describe quota -n dev
Name: test-resourcequota
Namespace: dev
Resource Used Hard
-------- ---- ----
limits.cpu 0 8
limits.memory 0 8000Mi
pods 0 4
requests.cpu 0 2
requests.memory 0 4000Mi
###创建大于4个的Pod副本数量
[root@k8s-master1 ~]# vi deployment-nginx.yaml
[root@k8s-master1 ~]# cat deployment-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx-1
name: nginx-1
spec:
replicas: 6
selector:
matchLabels:
app: nginx-1
template:
metadata:
creationTimestamp: null
labels:
app: nginx-1
spec:
containers:
- image: 10.245.4.88:8888/base-images/nginx
name: nginx-1
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 100Mi
cpu: 100m
imagePullSecrets:
- name: harbor-login
[root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yaml -n dev
deployment.apps/nginx-1 created
[root@k8s-master1 ~]# kubectl get pod -n dev
NAME READY STATUS RESTARTS AGE
nginx-1-6b68fc54b8-f8kxf 1/1 Running 0 8s
nginx-1-6b68fc54b8-lwd2h 1/1 Running 0 8s
nginx-1-6b68fc54b8-mq54t 1/1 Running 0 8s
nginx-1-6b68fc54b8-mvg9n 1/1 Running 0 8s
[root@k8s-master1 ~]# kubectl get deployment nginx-1 -n dev
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-1 4/6 4 4 24s ###得到了配额限制
[root@k8s-master1 ~]# kubectl describe quota -n dev
Name: test-resourcequota
Namespace: dev
Resource Used Hard
-------- ---- ----
limits.cpu 400m 8
limits.memory 400Mi 8000Mi
pods 4 4
requests.cpu 400m 2
requests.memory 400Mi 4000Mi
- 结论：当pod数量达到了配额限制后也是无法完成调度的
再再几点考虑
1、在上面条件都满足下，依然出现了node节点资源不足，这是必然的，因为团队会越来越大，那该怎么办呢
2、或者说一个node节点突然有个程序抽风，占用了大量资源，那对于上面的pod可是致命的打击，那应该怎么办呢
不用着急，k8s也想到了。
kubernetes对于pod的驱逐策略
kubernetes提供了pod eviction,Kube-controller-manager会周期性的检查所有node的状态，在某些情况下node资源不足的时候，需要将node上面的pod驱逐至其他节点，用于保障业务的正常运行
可创建的驱逐配置
--eviction-soft=memory.available<2Gi ###当内存小于2G的时候进行驱逐
--eviction-soft-grace-period=memory.available=2m ##当内存持续2分钟，结合上面的小于2G的时候进行驱逐
--eviction-hard=设定的条件，多条件用逗号分隔 ####当满足后面所有条件的时候立即进行驱逐
还有好多的条件，小编没有一一去测试，毕竟都深夜了，还请大家原谅，等小编有空的话，小编再一一进行测试验证
结束语
小编也是查了好多的资料和看了好多的视频，总结出来的，虽然说不是官方，但是也是按照视频或者文档，一步一步的进行验证下来的，如果有什么总结不是和合理的地方，还请大家多多指正，谢谢大家
本着学习和共同进步的目的，小编也是参照了好多的博客和视频来编写这个文章的，小编觉得写的较好的，都总结了下来，如果有什么侵犯或者不如意的地方，还请联系小编，进行文章的修改，谢谢