045.集群存储-CSI存储机制

一 csi存储机制

1.1 csi简介

kubernetes从1.9版本开始引入容器存储接口container storage interface（csi）机制，用于在kubernetes和外部存储系统之间建立一套标准的存储管理接口，通过该接口为容器提供存储服务。

1.2 csi的设计背景

kubernetes通过pv、pvc、storageclass已经提供了一种强大的基于插件的存储管理机制，但是各种存储插件提供的存储服务都是基于一种被称为“in-true”（树内）的方式提供的，这要求存储插件的代码必须被放进kubernetes的主干代码库中才能被kubernetes调用，属于紧耦合的开发模式。这种“in-tree”方式会带来一些问题：

• 存储插件的代码需要与kubernetes的代码放在同一代码库中，并与kubernetes的二进制文件共同发布；
• 存储插件代码的开发者必须遵循kubernetes的代码开发规范；
• 存储插件代码的开发者必须遵循kubernetes的发布流程，包括添加对kubernetes存储系统的支持和错误修复；
• kubernetes社区需要对存储插件的代码进行维护，包括审核、测试等工作；
• 存储插件代码中的问题可能会影响kubernetes组件的运行，并且很难排查问题；
• 存储插件代码与kubernetes的核心组件（kubelet和kubecontroller-manager）享有相同的系统特权权限，可能存在可靠性和安全性问题。

kubernetes已有的flexvolume插件机制试图通过为外部存储暴露一个基于可执行程序（exec）的api来解决这些问题。尽管它允许第三方存储提供商在kubernetes核心代码之外开发存储驱动，但仍然有两个问题没有得到很好的解决：

• 部署第三方驱动的可执行文件仍然需要宿主机的root权限，存在安全隐患；
• 存储插件在执行mount、attach这些操作时，通常需要在宿主机上安装一些第三方工具包和依赖库，使得部署过程更加复杂，例如部署ceph时需要安装rbd库，部署glusterfs时需要安装mount.glusterfs库，等等。

基于以上这些问题和考虑，kubernetes逐步推出与容器对接的存储接口标准，存储提供方只需要基于标准接口进行存储插件的实现，就能使用kubernetes的原生存储机制为容器提供存储服务。这套标准被称为csi（容器存储接口）。

在csi成为kubernetes的存储供应标准之后，存储提供方的代码就能和kubernetes代码彻底解耦，部署也与kubernetes核心组件分离，显然，存储插件的开发由提供方自行维护，就能为kubernetes用户提供更多的存储功能，也更加安全可靠。

基于csi的存储插件机制也被称为“out-of-tree”（树外）的服务提供方式，是未来kubernetes第三方存储插件的标准方案。

二 csi架构

2.1 csi存储组件/部署架构

kubernetescsi存储插件的关键组件和推荐的容器化部署架构：

其中主要包括两种组件：csi controller和csi node。

2.2 csi controller

csi controller的主要功能是提供存储服务视角对存储资源和存储卷进行管理和操作。在kubernetes中建议将其部署为单实例pod，可以使用statefulset或deployment控制器进行部署，设置副本数量为1，保证为一种存储插件只运行一个控制器实例。

在这个pod内部署两个容器：

• 与master（kube-controller-manager）通信的辅助sidecar容器。在sidecar容器内又可以包含external-attacher和external-provisioner两个容器，它们的功能分别如下。

• external-attacher：监控volumeattachment资源对象的变更，触发针对csi端点的controllerpublish和controllerunpublish操作。
• external-provisioner：监控persistentvolumeclaim资源对象的变更，触发针对csi端点的createvolume和deletevolume操作。
• csi driver存储驱动容器，由第三方存储提供商提供，需要实现上述接口。

这两个容器通过本地socket（unix domainsocket，uds），并使用gprc协议进行通信。

sidecar容器通过socket调用csi driver容器的csi接口，csi driver容器负责具体的存储卷操作。

2.3 csi node

csi node的主要功能是对主机（node）上的volume进行管理和操作。在kubernetes中建议将其部署为daemonset，在每个node上都运行一个pod。

在这个pod中部署以下两个容器：

• 与kubelet通信的辅助sidecar容器node-driver-registrar，主要功能是将存储驱动注册到kubelet中；
• csi driver存储驱动容器，由第三方存储提供商提供，主要功能是接收kubelet的调用，需要实现一系列与node相关的csi接口，例如nodepublishvolume接口（用于将volume挂载到容器内的目标路径）、nodeunpublishvolume接口（用于从容器中卸载volume），等等。

node-driver-registrar容器与kubelet通过node主机的一个hostpath目录下的unixsocket进行通信。csi driver容器与kubelet通过node主机的另一个hostpath目录下的unixsocket进行通信，同时需要将kubelet的工作目录（默认为/var/lib/kubelet）挂载给csidriver容器，用于为pod进行volume的管理操作（包括mount、umount等）。

三 csi插件使用实践

3.1 实验说明

以csi-hostpath插件为例，演示部署csi插件、用户使用csi插件提供的存储资源。

3.2 开启特性

设置kubernetes服务启动参数，为kube-apiserver、kubecontroller-manager和kubelet服务的启动参数添加。

[root@k8smaster01 ~]# vi /etc/kubernetes/manifests/kube-apiserver.yaml

……
    - --allow-privileged=true
    - --feature-gates=csipersistentvolume=true
    - --runtime-config=storage.k8s.io/v1alpha1=true
……

[root@k8smaster01 ~]# vi /etc/kubernetes/manifests/kube-controller-manager.yaml

……
    - --feature-gates=csipersistentvolume=true
……

[root@k8smaster01 ~]# vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

# note: this dropin only works with kubeadm and kubelet v1.11+
[service]
environment="kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --feature-gates=csipersistentvolume=true"
……

[root@k8smaster01 ~]# systemctl daemon-reload

[root@k8smaster01 ~]# systemctl restart kubelet.service

3.3 创建crd资源对象

创建csinodeinfo和csidriverregistry crd资源对象：

[root@k8smaster01 ~]# vi csidriver.yaml

apiversion: apiextensions.k8s.io/v1beta1
kind: customresourcedefinition
metadata:
  name: csidrivers.csi.storage.k8s.io
  labels:
    addonmanager.kubernetes.io/mode: reconcile
spec:
  group: csi.storage.k8s.io
  names:
    kind: csidriver
    plural: csidrivers
  scope: cluster
  validation:
    openapiv3schema:
      properties:
        spec:
          description: specification of the csi driver.
          properties:
            attachrequired:
              description: indicates this csi volume driver requires an attach operation,and that kubernetes should call attach and wait for any attach operationto complete before proceeding to mount.
              type: boolean
            podinfoonmountversion:
              description: indicates this csi volume driver requires additional pod
                information (like podname, poduid, etc.) during mount operations.
              type: string
  version: v1alpha1

[root@k8smaster01 ~]# vi csinodeinfo.yaml

apiversion: apiextensions.k8s.io/v1beta1
kind: customresourcedefinition
metadata:
  name: csinodeinfos.csi.storage.k8s.io
  labels:
    addonmanager.kubernetes.io/mode: reconcile
spec:
  group: csi.storage.k8s.io
  names:
    kind: csinodeinfo
    plural: csinodeinfos
  scope: cluster
  validation:
    openapiv3schema:
      properties:
        spec:
          description: specification of csinodeinfo
          properties:
            drivers:
              description: list of csi drivers running on the node and their specs.
              type: array
              items:
                properties:
                  name:
                    description: the csi driver that this object refers to.
                    type: string
                  nodeid:
                    description: the node from the driver point of view.
                    type: string
                  topologykeys:
                    description: list of keys supported by the driver.
                    items:
                      type: string
                    type: array
        status:
          description: status of csinodeinfo
          properties:
            drivers:
              description: list of csi drivers running on the node and their statuses.
              type: array
              items:
                properties:
                  name:
                    description: the csi driver that this object refers to.
                    type: string
                  available:
                    description: whether the csi driver is installed.
                    type: boolean
                  volumepluginmechanism:
                    description: indicates to external components the required mechanism
                      to use for any in-tree plugins replaced by this driver.
                    pattern: in-tree|csi
                    type: string
  version: v1alpha1

[root@k8smaster01 ~]# kubectl apply -f csidriver.yaml

[root@k8smaster01 ~]# kubectl apply -f csinodeinfo.yaml

3.4 创建相应rbac

[root@k8smaster01 ~]# git clone https://github.com/kubernetes-csi/drivers

[root@k8smaster01 ~]# cd drivers/deploy/hostpath/

[root@k8smaster01 hostpath]# vi csi-hostpath-attacher-rbac.yaml

---
apiversion: v1
kind: serviceaccount
metadata:
  name: csi-attacher
  # replace with non-default namespace name
  namespace: default
---
kind: clusterrole
apiversion: rbac.authorization.k8s.io/v1
metadata:
  name: external-attacher-runner
rules:
  - apigroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update"]
  - apigroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apigroups: ["csi.storage.k8s.io"]
    resources: ["csinodeinfos"]
    verbs: ["get", "list", "watch"]
  - apigroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
---
kind: clusterrolebinding
apiversion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-attacher-role
subjects:
  - kind: serviceaccount
    name: csi-attacher
    # replace with non-default namespace name
    namespace: default
roleref:
  kind: clusterrole
  name: external-attacher-runner
  apigroup: rbac.authorization.k8s.io
---
kind: role
apiversion: rbac.authorization.k8s.io/v1
metadata:
  # replace with non-default namespace name
  namespace: default
  name: external-attacher-cfg
rules:
- apigroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: rolebinding
apiversion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-attacher-role-cfg
  # replace with non-default namespace name
  namespace: default
subjects:
  - kind: serviceaccount
    name: csi-attacher
    # replace with non-default namespace name
    namespace: default
roleref:
  kind: role
  name: external-attacher-cfg
  apigroup: rbac.authorization.k8s.io

[root@k8smaster01 hostpath]# vi csi-hostpath-provisioner-rbac.yaml

---
apiversion: v1
kind: serviceaccount
metadata:
  name: csi-provisioner
  # replace with non-default namespace name
  namespace: default
---
kind: clusterrole
apiversion: rbac.authorization.k8s.io/v1
metadata:
  name: external-provisioner-runner
rules:
  - apigroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apigroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apigroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apigroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apigroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apigroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list"]
  - apigroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list"]
  - apigroups: ["csi.storage.k8s.io"]
    resources: ["csinodeinfos"]
    verbs: ["get", "list", "watch"]
  - apigroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
kind: clusterrolebinding
apiversion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-provisioner-role
subjects:
  - kind: serviceaccount
    name: csi-provisioner
    # replace with non-default namespace name
    namespace: default
roleref:
  kind: clusterrole
  name: external-provisioner-runner
  apigroup: rbac.authorization.k8s.io
---
kind: role
apiversion: rbac.authorization.k8s.io/v1
metadata:
  # replace with non-default namespace name
  namespace: default
  name: external-provisioner-cfg
rules:
- apigroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: rolebinding
apiversion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-provisioner-role-cfg
  # replace with non-default namespace name
  namespace: default
subjects:
  - kind: serviceaccount
    name: csi-provisioner
    # replace with non-default namespace name
    namespace: default
roleref:
  kind: role
  name: external-provisioner-cfg
  apigroup: rbac.authorization.k8s.io

[root@k8smaster01 hostpath]# vi csi-hostpathplugin-rbac.yaml

---
apiversion: v1
kind: serviceaccount
metadata:
  name: csi-driver-registrar
  # replace with non-default namespace name
  namespace: default
---
kind: clusterrole
apiversion: rbac.authorization.k8s.io/v1
metadata:
  name: driver-registrar-runner
rules:
  - apigroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  # the following permissions are only needed when running
  # driver-registrar without the --kubelet-registration-path
  # parameter, i.e. when using driver-registrar instead of
  # kubelet to update the csi.volume.kubernetes.io/nodeid
  # annotation. that mode of operation is going to be deprecated
  # and should not be used anymore, but is needed on older
  # kubernetes versions.
  # - apigroups: [""]
  #   resources: ["nodes"]
  #   verbs: ["get", "update", "patch"]
---
kind: clusterrolebinding
apiversion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-driver-registrar-role
subjects:
  - kind: serviceaccount
    name: csi-driver-registrar
    # replace with non-default namespace name
    namespace: default
roleref:
  kind: clusterrole
  name: driver-registrar-runner
  apigroup: rbac.authorization.k8s.io

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-attacher-rbac.yaml

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-provisioner-rbac.yaml

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpathplugin-rbac.yaml

3.5 正式部署

[root@k8smaster01 ~]# cd drivers/deploy/hostpath/

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-attacher.yaml

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-provisioner.yaml

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpathplugin.yaml

提示：如上相应yaml建议修改镜像源为国内：

gcr.io ----> gcr.azk8s.cn （国内）

quay.io ----> quay.azk8s.cn （国内）

四 测试使用

4.1 确认验证

[root@k8smaster01 ~]# kubectl get pods

4.2 创建storageclass

[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-storageclass.yaml

apiversion: storage.k8s.io/v1
kind: storageclass
metadata:
  name: csi-hostpath-sc
provisioner: csi-hostpath
reclaimpolicy: delete
volumebindingmode: immediate

[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-storageclass.yaml

4.3 创建pvc

[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-pvc.yaml

apiversion: v1
kind: persistentvolumeclaim
metadata:
  name: csi-pvc
spec:
  accessmodes:
  - readwriteonce
  resources:
    requests:
      storage: 1gi
  storageclassname: csi-hostpath-sc

[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-pvc.yaml

[root@k8smaster01 ~]# kubectl get pvc

[root@k8smaster01 ~]# kubectl get pv

4.4 创建应用

[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-app.yaml

kind: pod
apiversion: v1
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-frontend
      image: busybox
      volumemounts:
      - mountpath: "/data"
        name: my-csi-volume
      command: [ "sleep", "1000000" ]
  volumes:
    - name: my-csi-volume
      persistentvolumeclaim:
        claimname: csi-pvc

[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-app.yaml

[root@k8smaster01 ~]# kubectl get pods

提示：更多csi插件示例参考：https://feisky.gitbooks.io/kubernetes/plugins/csi.html。

csi官方文档：https://kubernetes-csi.github.io/docs/