kubernetes资源配额管理

程序员文章站 2024-03-19 14:06:04

...

参考：https://kubernetes.io/docs/concepts/policy/resource-quotas/

当集群有多个用户或者团队时，一个重要的问题就是资源的公平分配。

资源配置就是管理员用来解决类似问题的工具。

通过定义ResourceQuota对象，约束一个名称空间可以消耗的系统资源的总数。它可以限制一个名称空间内可以创建的某种类型对象的数量，也可以限制名称空间所能消耗的低层资源的总数。

资源配额按如下方式工作：

不同teams工作在不同namespace下，目前这种个约束需要用户自愿遵守，但支持通过ACLs强制遵守此约束。
由管理员为每个namespace创建一个或者多个ResourceQuota。
用户在namespace下创建各种资源，配额系统追踪namespace下各种资源的使用量确保不会超过ResourceQuota规定的硬性数量。
如果在创建、更新资源时违反了资源配额规定，apiserver返回403应答并在正文中解决原因。
如果名称空间中有对计算资源如cpu、memory等的配额限制，则用户在创建pod时必需指明对这些资源的最小需求与最大限制，否则系统创建pod。提示：用LimitRanger admission controller(入口管理器)强制为没有指定对资源需求的pod设定默认值。

你可能会创建如下资源配额规则：

集群有32 GiB RAM、16 cores，允许team A使用20 GiB RAM、10 cores，允许B使用10 GiB RAM、4 cores，剩下2GiB RAM、2 cores保留。
限制“testing” namespace使用1core、1GiB RAM。让“production” namespace没有配额限制。

发系统中的资源总额少于系统中所有资源配额的总时，会发生争用情况，目前的处理措施是先到先得。

配额对已经创建的资源没有影响。

允许资源配额管理

默认安装，apiserver的标志--enable-admission-plugins的值包含ResourceQuota，资源配额管理默认打开。

如果某个namespace包含资源配额条目，那么无法apiserver标志是否打开，针对这个namespace的资源配额管理都会被打开。

计算资源配额管理

可以为namespace设置计算资源配额，支持如下类型：

Resource Name	Description
`cpu`	Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value.
`limits.cpu`	Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.
`limits.memory`	Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value.
`memory`	Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value.
`requests.cpu`	Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value.
`requests.memory`	Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value.

扩展资源资源配额

允许对扩展资源使用资源配额管理。方式与系统内置资源类型有所不同。对于扩展资源类型的限制，只需要指定limit值，无需指定request值。例如，以GPU资源为例，资源的名称是nvidia.com/gpu，限制namespace对此资源的配额为4，那么quota的定义如下：

requests.nvidia.com/gpu: 4

存储资源配额

可以为namespace指定存储资源配额，进一步的话可以基于存储资源的种类进行配额管理。

esource Name	Description
`requests.storage`	Across all persistent volume claims, the sum of storage requests cannot exceed this value.
`persistentvolumeclaims`	The total number of persistent volume claims that can exist in the namespace.
`<storage-class-name>.storageclass.storage.k8s.io/requests.storage`	Across all persistent volume claims associated with the storage-class-name, the sum of storage requests cannot exceed this value.
`<storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaims`	Across all persistent volume claims associated with the storage-class-name, the total number of persistent volume claims that can exist in the namespace.

如下分别对金牌与铜牌两种存储做配额管理：

gold.storageclass.storage.k8s.io/requests.storage: 500Gi
bronze.storageclass.storage.k8s.io/requests.storage: 100G

从1.8版本开始，作为测试我支持对临时存储的配额管理：

Resource Name	Description
`requests.ephemeral-storage`	Across all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value.
`limits.ephemeral-storage`	Across all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value.

对象数量配额

从1.9版本开始，支持所有标准对象类型的配额管理，语法如下：

count/<resource>.<group>

以下是可以数量配额管理的资源类型：

count/persistentvolumeclaims
count/services
count/secrets
count/configmaps
count/replicationcontrollers
count/deployments.apps
count/replicasets.apps
count/statefulsets.apps
count/jobs.batch
count/cronjobs.batch
count/deployments.extensions

对象数量配额可以防止某种类型的对象太多而引起的系统整体性能的下降。以下是支持的配额项：

Resource Name	Description
`configmaps`	The total number of config maps that can exist in the namespace.
`persistentvolumeclaims`	The total number of persistent volume claims that can exist in the namespace.
`pods`	The total number of pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if `.status.phase in (Failed, Succeeded)` is true.
`replicationcontrollers`	The total number of replication controllers that can exist in the namespace.
`resourcequotas`	The total number of resource quotas that can exist in the namespace.
`services`	The total number of services that can exist in the namespace.
`services.loadbalancers`	The total number of services of type load balancer that can exist in the namespace.
`services.nodeports`	The total number of services of type node port that can exist in the namespace.
`secrets`	The total number of secrets that can exist in the namespace.

pod数量配额为限制namespace中所有没有终止的pod的数量，也许管理员想通过这种配额限制用户在namespace下创建过多的小pod从而用光可用的pod网络地址。

配额作用范围

每个配额可以与某个scope相关联，配额只度量与其范围相符的资源使用情况，只限制与其范围相符的资源使用。以下是支持的scope：

Scope	Description
`Terminating`	Match pods where `.spec.activeDeadlineSeconds >= 0`
`NotTerminating`	Match pods where `.spec.activeDeadlineSeconds is nil`
`BestEffort`	Match pods that have best effort quality of service.
`NotBestEffort`	Match pods that do not have best effort quality of service.

BestEffort类型的scope只支持pod类型配额。

erminating, NotTerminating, and NotBestEffort类型的scope支持如下类型的配额：

cpu
limits.cpu
limits.memory
memory
pods
requests.cpu
requests.memory

基于优先级的资源配额

首先在创建pod时可以指定优先级，pod的优先级不同，它的配限制也应该不同，比如优无级的pod应该有更严格的配额限制，反之则应该更宽松。这一点可以通过为配额指定scopeSelector字段实现。以下是示例，假设如下场景：

Pod有三种优先级：“low”, “medium”, “high”。
为每种优先级设定不同的配额。

具体步骤如下：

１.将如下内容保存到quota.yml文件：

```yaml apiVersion: v1 kind: List items:

apiVersion: v1 kind: ResourceQuota metadata: name: pods-high spec: hard: cpu: “1000” memory: 200Gi pods: “10” scopeSelector: matchExpressions: - operator : In scopeName: PriorityClass values: [“high”]
apiVersion: v1 kind: ResourceQuota metadata: name: pods-medium spec: hard: cpu: “10” memory: 20Gi pods: “10” scopeSelector: matchExpressions: - operator : In scopeName: PriorityClass values: [“medium”]
apiVersion: v1 kind: ResourceQuota metadata: name: pods-low spec: hard: cpu: “5” memory: 10Gi pods: “10” scopeSelector: matchExpressions: - operator : In scopeName: PriorityClass values: [“low”] ```

２.创建三个quota对象

 kubectl create -f ./quota.yml`

  resourcequota/pods-high created
  resourcequota/pods-medium created
  resourcequota/pods-low created

３.确认使用掉的配额为"0"

kubectl describe quota

  Name:       pods-high
  Namespace:  default
  Resource    Used  Hard
  --------    ----  ----
  cpu         0     1k
  memory      0     200Gi
  pods        0     10


  Name:       pods-low
  Namespace:  default
  Resource    Used  Hard
  --------    ----  ----
  cpu         0     5
  memory      0     10Gi
  pods        0     10


  Name:       pods-medium
  Namespace:  default
  Resource    Used  Hard
  --------    ----  ----
  cpu         0     10
  memory      0     20Gi
  pods        0     10

４.创建high优先级的pod，保存如下内容到high-priority-pod.yml文件

apiVersion: v1
  kind: Pod
  metadata:
    name: high-priority
  spec:
    containers:
    - name: high-priority
      image: ubuntu
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo hello; sleep 10;done"]
      resources:
        requests:
          memory: "10Gi"
          cpu: "500m"
        limits:
          memory: "10Gi"
          cpu: "500m"
    priorityClassName: high

创建对象：

kubectl create -f ./high-priority-pod.yml

5.确认pods-high中资源使用情况，说明对pod生效的配额是pods-high。

kubectl describe quota

  Name:       pods-high
  Namespace:  default
  Resource    Used  Hard
  --------    ----  ----
  cpu         500m  1k
  memory      10Gi  200Gi
  pods        1     10


  Name:       pods-low
  Namespace:  default
  Resource    Used  Hard
  --------    ----  ----
  cpu         0     5
  memory      0     10Gi
  pods        0     10


  Name:       pods-medium
  Namespace:  default
  Resource    Used  Hard
  --------    ----  ----
  cpu         0     10
  memory      0     20Gi
  pods        0     10

scopeSelector支持的操作符：

In
NotIn
Exist
DoesNotExist

Requests vs Limits

这两个是在创建资源时为其指定的属性。Requests表示创建对象所必需的资源数值。Limits表示对象可以占用的最大资源上限。

查看设置配额

kubectl支持创建、更新、查看配额。

kubectl create namespace myspace

cat <<EOF > compute-resources.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    pods: "4"
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
    requests.nvidia.com/gpu: 4
EOF
kubectl create -f ./compute-resources.yaml --namespace=myspace

cat <<EOF > object-counts.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
spec:
  hard:
    configmaps: "10"
    persistentvolumeclaims: "4"
    replicationcontrollers: "20"
    secrets: "10"
    services: "10"
    services.loadbalancers: "2"
EOF
kubectl create -f ./object-counts.yaml --namespace=myspace

kubectl get quota --namespace=myspace
NAME                    AGE
compute-resources       30s
object-counts           32s

kubectl describe quota compute-resources --namespace=myspace
Name:                    compute-resources
Namespace:               myspace
Resource                 Used  Hard
--------                 ----  ----
limits.cpu               0     2
limits.memory            0     2Gi
pods                     0     4
requests.cpu             0     1
requests.memory          0     1Gi
requests.nvidia.com/gpu  0     4


kubectl describe quota object-counts --namespace=myspace
Name:                   object-counts
Namespace:              myspace
Resource                Used    Hard
--------                ----    ----
configmaps              0       10
persistentvolumeclaims  0       4
replicationcontrollers  0       20
secrets                 1       10
services                0       10
services.loadbalancers  0       2

kubectl也支持标准类型资源的数据配额。

kubectl create namespace myspace

kubectl create quota test --hard=count/deployments.extensions=2,count/replicasets.extensions=4,count/pods=3,count/secrets=4 --namespace=myspace

kubectl run nginx --image=nginx --replicas=2 --namespace=myspace

kubectl describe quota --namespace=myspace
Name:                         test
Namespace:                    myspace
Resource                      Used  Hard
--------                      ----  ----
count/deployments.extensions  1     2
count/pods                    2     3
count/replicasets.extensions  1     4
count/secrets                 1     4

Quota and Cluster Capacity

配额用绝对值表示，这一点与集群容量不同。比如集群中新加一个node，此时集群的容量变大，但是namespace下的配额不会相应的放宽。如果想要配额与集群容量联动的话，可以考虑扩展一个控制器，监控集群容量，再根据设置好的规则动态调整配额。

Limit Priority Class consumption by default

对于系统中很高的优先级，比如“cluster-services”，管理员想限制系统中某些namespace创建这种优先级的pod，也就是说只允许特定的namespace能创建这么高优先级的pod，实现方法如下。

apiserver指定如下选项--admission-control-config-file，其值为文件路径，文件内容如下：

$ cat admission_config_file.yml
apiVersion: apiserver.k8s.io/v1alpha1
kind: AdmissionConfiguration
plugins:
- name: "ResourceQuota"
  configuration:
    apiVersion: resourcequota.admission.k8s.io/v1alpha1
    kind: Configuration
    limitedResources:
    - resource: pods
    matchScopes:
    - operator : In
      scopeName: PriorityClass
      values: ["cluster-services"]

如此，cluster-services优先级的pod只能在包含如下quota的namespace中创建：

 scopeSelector:
      matchExpressions:
      - operator : In
        scopeName: PriorityClass
        values: ["cluster-services"]

注意：scopeSelector测试特性，使用之前先打开特性开关ResourceQuotaScopeSelectors。