k8s动态使用glusterfs存储
程序员文章站
2022-08-31 19:30:42
k8s中运行Heketi服务
在使用kismatic安装k8s和glusterfs后,需要配置k8s默认动态管理(provisioning)使用glusterfs,以下内容是实现这样的目的。...
k8s中运行Heketi服务
在使用kismatic安装k8s和glusterfs后,需要配置k8s默认动态管理(provisioning)使用glusterfs,以下内容是实现这样的目的。
Hekeli:glusterfs的restful端点,k8s向它发送命令来使用glusterfs。
Hekeli cli安装
下载Hekeli cli工具,安装在$PATH路径下
Hekeli 服务端安装在k8s环境
在所有的glusterfs节点,创建hekeli的数据库存储目录、ssh免密码登录文件目录。
mkdir -p /data/heketi/{db,.ssh} && chmod 700 /data/heketi/.ssh
生成ssh密钥
ssh-keygen -t rsa -b 2048 -f /data/heketi/.ssh/id_rsa for NODE in node1 node2 node3; do scp -r /data/heketi/.ssh root@${NODE}:/data/heketi; done for NODE in node1 node2 node3; do ssh-copy-id -i /data/heketi/.ssh/id_rsa.pub root@${NODE} ; done
k8s运行heketi的服务
kubectl apply -f heketi-secret.yaml -f heketi-deployment.json
heketi-secret.yaml文件内容
apiVersion: v1 kind: Secret metadata: name: heketi-secret namespace: default data: # base64 encoded password. E.g.: echo -n "password" | base64 key: cGFzc3dvcmQ= type: kubernetes.io/glusterfs
heketi-deployment.json文件内容
{ "kind": "List", "apiVersion": "v1", "items": [ { "kind": "Service", "apiVersion": "v1", "metadata": { "name": "heketi", "labels": { "glusterfs": "heketi-service", "deploy-heketi": "support" }, "annotations": { "description": "Exposes Heketi Service", "tags": "kubernetes,k8s,heketi", "traefik.backend.loadbalancer": "wrr", "traefik.backend.weight": "10", "traefik.enable": "true", "traefik.frontend.entryPoints": "http,https", "traefik.frontend.rule": "Host:heketi-api.example.com", "traefik.tags": "kubernetes" } }, "spec": { "type": "NodePort", "selector": { "name": "heketi" }, "ports": [ { "name": "heketi", "port": 8080, "targetPort": 8080, “nodePort”: 30944 } ] } }, { "kind": "Deployment", "apiVersion": "apps/v1beta1", "metadata": { "name": "heketi", "labels": { "glusterfs": "heketi-deployment" }, "annotations": { "description": "Defines how to deploy Heketi" } }, "spec": { "replicas": 1, "strategy": { "rollingUpdate": { "maxSurge": 0, "maxUnavailable": 1 }, "type": "RollingUpdate" }, "template": { "metadata": { "name": "heketi", "labels": { "name": "heketi", "glusterfs": "heketi-pod" } }, "spec": { "terminationGracePeriodSeconds": 0, "nodeSelector": { "storagenode": "glusterfs" }, "containers": [ { "image": "heketi/heketi:5", "imagePullPolicy": "Always", "name": "heketi", "env": [ { "name": "HEKETI_EXECUTOR", "value": "ssh" }, { "name": "HEKETI_SSH_USER", "value": "root" }, { "name": "HEKETI_SSH_PORT", "value": "22" }, { "name": "HEKETI_SSH_KEYFILE", "value": "/root/.ssh/id_rsa" }, { "name": "HEKETI_ADMIN_KEY", "valueFrom": { "secretKeyRef": { "name": "heketi-secret", "key": "key" } } } ], "ports": [ { "containerPort": 8080 } ], "volumeMounts": [ { "name": "heketi-ssh-key", "mountPath": "/root/.ssh" }, { "name": "heketi-db", "mountPath": "/var/lib/heketi" } ], "readinessProbe": { "timeoutSeconds": 3, "initialDelaySeconds": 3, "httpGet": { "path": "/hello", "port": 8080 } }, "livenessProbe": { "timeoutSeconds": 3, "initialDelaySeconds": 15, "httpGet": { "path": "/hello", "port": 8080 } } } ], "volumes": [ { "name": "heketi-ssh-key", "hostPath": { "path": "/data/heketi/.ssh" } }, { "name": "heketi-db", "hostPath": { "path": "/data/heketi/db" } } ] } } } } ] }
这里配置Heketi的service在NodeIP:30944上
通过命令检查heketi服务
curl -s http://nodeIP:30944/hello
通过cli连接
heketi-cli --user admin --secret password --server http://node1:30944 cluster list
导入glusterfs集群拓扑(topology)信息
heketi-cli --user admin --secret password --server http://node1:30944 topology load --json heketi-topology.json
heketi-topology.json的文件内容为
{ "clusters": [ { "nodes": [ { "node": { "hostnames": { "manage": [ "192.168.122.112" ], "storage": [ "192.168.122.112" ] }, "zone": 1 }, "devices": [ "/dev/vdb" ] } ] } ] }
有关glusterfs集群的topology的配置
使用以下命令获得cluster id
heketi-cli --user admin --secret password --server http://192.168.122.112:30944 cluster list
k8s使用glusterfs
k8s上定义glusterds的default storageclass,然后通过pvc来动态使用glusterfs存储
kubectl apply -f heketi-storageclass.yaml
heketi-storageclass.yaml的内容如下
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: slow annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/glusterfs parameters: resturl: "http://192.168.122.112:30944" clusterid: "e4b83af0a6260975ed1001ea3e89a0ad" restuser: "admin" secretNamespace: "default" secretName: "heketi-secret" volumetype: "none"
kubectl apply -f heketi-pvc.yaml kubectl get pvc test-claim kubectl get pv
heketi-pvc.yaml的文件内容
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: test-claim spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi
使用命令查看glusterfs中的volume
heketi-cli --user admin --secret password --server http://192.168.122.112:30944 volume list
创建pod使用pvc
kubectl apply -f nginx-pod.yaml
nginx-pod.yaml的内容
kind: Pod apiVersion: v1 metadata: name: nginx-with-pv spec: containers: - name: frontend image: nginx:stable-alpine volumeMounts: - mountPath: "/usr/share/nginx/html" name: pv nodeSelector: storagenode: "glusterfs" volumes: - name: pv persistentVolumeClaim: claimName: test-claim
上一篇: 老版本百度网盘不让QQ登录了