SUSE Ceph iSCSI 网关管理 - Storage6
程序员文章站
2022-04-09 21:13:27
iSCSI网关集成了Ceph存储和iSCSI标准，以提供一个高可用性(HA) iSCSI目标，该目标将RADOS块设备(RBD)映像导出为SCSI磁盘。iSCSI协议允许客户机 (initiator) 通过TCP/IP网络向SCSI存储设备( targets )发送SCSI命令。这允许异构客户机访问 ......
iscsi网关集成了ceph存储和iscsi标准，以提供一个高可用性(ha) iscsi目标，该目标将rados块设备(rbd)映像导出为scsi磁盘。iscsi协议允许客户机 (initiator) 通过tcp/ip网络向scsi存储设备( targets )发送scsi命令。这允许异构客户机访问ceph存储集群。
每个iscsi网关运行linux io目标内核子系统(lio)，以提供iscsi协议支持。lio利用用户空间通过( tcmu ) 与ceph的librbd库交互，并向iscsi客户机暴露rbd镜像。使用ceph的iscsi网关，可以有效地运行一个完全集成的块存储基础设施，它具有传统存储区域网络(san)的所有特性和优点。
rbd 作为 vmware esxi datastore 是否支持？
（1）目前来说，rbd是不支持datastore形式。
（2）iscsi 是支持 datastore 这种方式，可以作为vmware esxi 虚拟机提供存储功能，性价比非常不错的选择。
1、创建池和镜像
（1）创建池
# ceph osd pool create iscsi-images 128 128 replicated # ceph osd pool application enable iscsi-images rbd
（2）创建images
# rbd --pool iscsi-images create --size=2048 'iscsi-gateway-image001' # rbd --pool iscsi-images create --size=4096 'iscsi-gateway-image002' # rbd --pool iscsi-images create --size=2048 'iscsi-gateway-image003' # rbd --pool iscsi-images create --size=4096 'iscsi-gateway-image004'
（3）显示images
# rbd ls -p iscsi-images iscsi-gateway-image001 iscsi-gateway-image002 iscsi-gateway-image003 iscsi-gateway-image004
2、deepsea 方式安装iscsi网关
（1）node001 和 node002节点上安装，编辑policy.cfg 文件
vim /srv/pillar/ceph/proposals/policy.cfg ...... # igw role-igw/cluster/node00[1-2]*.sls ......
（2）运行 stage 2 和 stage 4
# salt-run state.orch ceph.stage.2
# salt 'node001*' pillar.items
public_network:
192.168.2.0/24
roles:
- mon
- mgr
- storage
- igw
time_server:
admin.example.com
# salt-run state.orch ceph.stage.4
3、手动方式安装iscsi网关
（1）node003 节点安装 iscsi 软件包
# zypper -n in -t pattern ceph_iscsi # zypper -n in tcmu-runner tcmu-runner-handler-rbd \ ceph-iscsi patterns-ses-ceph_iscsi python3-flask python3-click python3-configshell-fb \ python3-itsdangerous python3-netifaces python3-rtslib-fb \ python3-targetcli-fb python3-urwid targetcli-fb-common
（2）admin节点创建key,并复制到 node003
# ceph auth add client.igw.node003 mon 'allow *' osd 'allow *' mgr 'allow r'
# ceph auth get client.igw.node003
client.igw.node003
key: aqc0eotdaaaaabaaszrzh9keo0v0wtftcw9ahq==
caps: [mgr] allow r
caps: [mon] allow *
caps: [osd] allow *
# ceph auth get client.igw.node003 >> /etc/ceph/ceph.client.igw.node003.keyring # scp /etc/ceph/ceph.client.igw.node003.keyring node003:/etc/ceph
（3）node003 节点启动服务
# systemctl start tcmu-runner.service # systemctl enable tcmu-runner.service
（4）node003 节点创建配置文件
# vim /etc/ceph/iscsi-gateway.cfg [config] cluster_client_name = client.igw.node003 pool = iscsi-images trusted_ip_list = 192.168.2.42,192.168.2.40,192.168.2.41 minimum_gateways = 1 fqdn_enabled=true # additional api configuration options are as follows, defaults shown. api_port = 5000 api_user = admin api_password = admin api_secure = false # log level logger_level = warning
（5）启动 rbd target 服务
# systemctl start rbd-target-api.service # systemctl enable rbd-target-api.service
（6）显示配置信息
# gwcli info http mode : http rest api port : 5000 local endpoint : http://localhost:5000/api local ceph cluster : ceph 2ndary api ip's : 192.168.2.42,192.168.2.40,192.168.2.41
# gwcli ls o- / ...................................................................... [...] o- cluster ...................................................... [clusters: 1] | o- ceph ......................................................... [health_ok] | o- pools ....................................................... [pools: 1] | | o- iscsi-images ........ [(x3), commit: 0.00y/15718656k (0%), used: 192k] | o- topology ............................................. [osds: 6,mons: 3] o- disks .................................................... [0.00y, disks: 0] o- iscsi-targets ............................ [discoveryauth: none, targets: 0]
4、dashboard 添加 iscsi 网关
（1）admin节点上，查看 dashboard iscsi 网关
admin:~ # ceph dashboard iscsi-gateway-list
{"gateways": {"node002.example.com": {"service_url": "http://admin:admin@192.168.2.41:5000"},
"node001.example.com": {"service_url": "http://admin:admin@192.168.2.40:5000"}}}
（2）添加 iscsi 网关
# ceph dashboard iscsi-gateway-add http://admin:admin@192.168.2.42:5000
# ceph dashboard iscsi-gateway-list
{"gateways": {"node002.example.com": {"service_url": "http://admin:admin@192.168.2.41:5000"},
"node001.example.com": {"service_url": "http://admin:admin@192.168.2.40:5000"},
"node003.example.com": {"service_url": "http://admin:admin@192.168.2.42:5000"}}}
（3）登陆 dashboard 查看 iscsi 网关
5、export rbd images via iscsi
（1）创建 iscsi target name
# gwcli gwcli > /> cd /iscsi-targets gwcli > /iscsi-targets> create iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01
（2）添加 iscsi 网关
gwcli > /iscsi-targets> cd iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/gateways /iscsi-target...tvol/gateways> create node001.example.com 172.200.50.40 /iscsi-target...tvol/gateways> create node002.example.com 172.200.50.41 /iscsi-target...tvol/gateways> create node003.example.com 172.200.50.42
/iscsi-target...ay01/gateways> ls o- gateways ......................................................... [up: 3/3, portals: 3] o- node001.example.com ............................................. [172.200.50.40 (up)] o- node002.example.com ............................................. [172.200.50.41 (up)] o- node003.example.com ............................................. [172.200.50.42 (up)]
注意：安装主机名来定义
/iscsi-target...tvol/gateways> create node002 172.200.50.41 the first gateway defined must be the local machine
（3）添加 rbd 镜像
/iscsi-target...tvol/gateways> cd /disks /disks> attach iscsi-images/iscsi-gateway-image001 /disks> attach iscsi-images/iscsi-gateway-image002
（4）target 和 rbd 镜像建立映射关系
/disks> cd /iscsi-targets/iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/disks /iscsi-target...teway01/disks> add iscsi-images/iscsi-gateway-image001 /iscsi-target...teway01/disks> add iscsi-images/iscsi-gateway-image002
（5）设置不验证
gwcli > /> cd /iscsi-targets/iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01/hosts /iscsi-target...teway01/hosts> auth disable_acl /iscsi-target...teway01/hosts> exit
（6）查看配置信息
node001:~ # gwcli ls
o- / ............................................................................... [...]
o- cluster ............................................................... [clusters: 1]
| o- ceph .................................................................. [health_ok]
| o- pools ................................................................ [pools: 1]
| | o- iscsi-images .................. [(x3), commit: 6g/15717248k (40%), used: 1152k]
| o- topology ...................................................... [osds: 6,mons: 3]
o- disks ................................................................ [6g, disks: 2]
| o- iscsi-images .................................................. [iscsi-images (6g)]
| o- iscsi-gateway-image001 ............... [iscsi-images/iscsi-gateway-image001 (2g)]
| o- iscsi-gateway-image002 ............... [iscsi-images/iscsi-gateway-image002 (4g)]
o- iscsi-targets ..................................... [discoveryauth: none, targets: 1]
o- iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01 .............. [gateways: 3]
o- disks ................................................................ [disks: 2]
| o- iscsi-images/iscsi-gateway-image001 .............. [owner: node001.example.com]
| o- iscsi-images/iscsi-gateway-image002 .............. [owner: node002.example.com]
o- gateways .................................................. [up: 3/3, portals: 3]
| o- node001.example.com ...................................... [172.200.50.40 (up)]
| o- node002.example.com ...................................... [172.200.50.41 (up)]
| o- node003.example.com ...................................... [172.200.50.42 (up)]
o- host-groups ........................................................ [groups : 0]
o- hosts .................................................... [hosts: 0: auth: none]
6、使用 dashboard 界面输出 rbd images
（1）添加 iscsi target
（2）编写 target iqn，并且添加镜像 portals 和 images
（3）查看新添加 iscsi target 信息
7、linux 客户端访问
（1）启动 iscsid 服务
- sles or rhel
# systemctl start iscsid.service # systemctl enable iscsid.service
- debian or ubuntu
# systemctl start open-iscsi
（2）发现和连接 targets
# iscsiadm -m discovery -t st -p 172.200.50.40 172.200.50.40:3260,1 iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01 172.200.50.41:3260,2 iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01 172.200.50.42:3260,3 iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway01 172.200.50.40:3260,1 iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway02 172.200.50.41:3260,2 iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway02 172.200.50.42:3260,3 iqn.2019-10.com.suse-iscsi.iscsi01.x86:iscsi-gateway02
（3）登录target
# iscsiadm -m node -p 172.200.50.40 --login # iscsiadm -m node -p 172.200.50.41 --login # iscsiadm -m node -p 172.200.50.42 --login
# lsblk name maj:min rm size ro type mountpoint sda 8:0 0 25g 0 disk ├─sda1 8:1 0 509m 0 part /boot └─sda2 8:2 0 24.5g 0 part ├─vg00-lvswap 254:0 0 2g 0 lvm [swap] └─vg00-lvroot 254:1 0 122.5g 0 lvm / sdb 8:16 0 100g 0 disk └─vg00-lvroot 254:1 0 122.5g 0 lvm / sdc 8:32 0 2g 0 disk sdd 8:48 0 2g 0 disk sde 8:64 0 4g 0 disk sdf 8:80 0 4g 0 disk sdg 8:96 0 2g 0 disk sdh 8:112 0 4g 0 disk sdi 8:128 0 2g 0 disk sdj 8:144 0 4g 0 disk sdk 8:160 0 2g 0 disk sdl 8:176 0 2g 0 disk sdm 8:192 0 4g 0 disk sdn 8:208 0 4g 0 disk
（4）如果系统上已安装 lsscsi 实用程序，您可以使用它来枚举系统上可用的 scsi 设备：
# lsscsi [1:0:0:0] cd/dvd necvmwar vmware sata cd01 1.00 /dev/sr0 [30:0:0:0] disk vmware, vmware virtual s 1.0 /dev/sda [30:0:1:0] disk vmware, vmware virtual s 1.0 /dev/sdb [33:0:0:0] disk suse rbd 4.0 /dev/sdc [33:0:0:1] disk suse rbd 4.0 /dev/sde [34:0:0:2] disk suse rbd 4.0 /dev/sdd [34:0:0:3] disk suse rbd 4.0 /dev/sdf [35:0:0:0] disk suse rbd 4.0 /dev/sdg [35:0:0:1] disk suse rbd 4.0 /dev/sdh [36:0:0:2] disk suse rbd 4.0 /dev/sdi [36:0:0:3] disk suse rbd 4.0 /dev/sdj [37:0:0:0] disk suse rbd 4.0 /dev/sdk [37:0:0:1] disk suse rbd 4.0 /dev/sdm [38:0:0:2] disk suse rbd 4.0 /dev/sdl [38:0:0:3] disk suse rbd 4.0 /dev/sdn
（5）多路径设置
# zypper in multipath-tools # modprobe dm-multipath path # systemctl start multipathd.service # systemctl enable multipathd.service
# multipath -ll 36001405863b0b3975c54c5f8d1ce0e01 dm-3 suse,rbd size=4.0g features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 35:0:0:1 sdh 8:112 active ready running <=== 单条链路 active `-+- policy='service-time 0' prio=10 status=enabled |- 33:0:0:1 sde 8:64 active ready running `- 37:0:0:1 sdm 8:192 active ready running 3600140529260bf41c294075beede0c21 dm-2 suse,rbd size=2.0g features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 33:0:0:0 sdc 8:32 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 35:0:0:0 sdg 8:96 active ready running `- 37:0:0:0 sdk 8:160 active ready running 360014055d00387c82104d338e81589cb dm-4 suse,rbd size=2.0g features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 38:0:0:2 sdl 8:176 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 34:0:0:2 sdd 8:48 active ready running `- 36:0:0:2 sdi 8:128 active ready running 3600140522ec3f9612b64b45aa3e72d9c dm-5 suse,rbd size=4.0g features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 34:0:0:3 sdf 8:80 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 36:0:0:3 sdj 8:144 active ready running `- 38:0:0:3 sdn 8:208 active ready running
（5）编辑多路径配置文件
# vim /etc/multipath.conf
defaults {
user_friendly_names yes
}
devices {
device {
vendor "(lio-org|suse)"
product "rbd"
path_grouping_policy "multibus" # 所有有效路径在一个优先组群中
path_checker "tur" # 在设备中执行 test unit ready 命令。
features "0"
hardware_handler "1 alua" # 在切换路径组群或者处理 i/o 错误时用来执行硬件具体动作的模块。
prio "alua"
failback "immediate"
rr_weight "uniform" # 所有路径都有相同的加权
no_path_retry 12 # 路径故障后，重试12次，每次5秒
rr_min_io 100 # 指定切换到当前路径组的下一个路径前路由到该路径的 i/o 请求数。
}
}
# systemctl stop multipathd.service # systemctl start multipathd.service
（6）查看多路径状态
# multipath -ll mpathd (3600140522ec3f9612b64b45aa3e72d9c) dm-5 suse,rbd size=4.0g features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=23 status=active |- 34:0:0:3 sdf 8:80 active ready running <=== 多条链路 active |- 36:0:0:3 sdj 8:144 active ready running `- 38:0:0:3 sdn 8:208 active ready running mpathc (360014055d00387c82104d338e81589cb) dm-4 suse,rbd size=2.0g features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=23 status=active |- 34:0:0:2 sdd 8:48 active ready running |- 36:0:0:2 sdi 8:128 active ready running `- 38:0:0:2 sdl 8:176 active ready running mpathb (36001405863b0b3975c54c5f8d1ce0e01) dm-3 suse,rbd size=4.0g features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=23 status=active |- 33:0:0:1 sde 8:64 active ready running |- 35:0:0:1 sdh 8:112 active ready running `- 37:0:0:1 sdm 8:192 active ready running mpatha (3600140529260bf41c294075beede0c21) dm-2 suse,rbd size=2.0g features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=23 status=active |- 33:0:0:0 sdc 8:32 active ready running |- 35:0:0:0 sdg 8:96 active ready running `- 37:0:0:0 sdk 8:160 active ready running
（7）显示当前的device mapper的信息
# dmsetup ls --tree mpathd (254:5) ├─ (8:208) ├─ (8:144) └─ (8:80) mpathc (254:4) ├─ (8:176) ├─ (8:128) └─ (8:48) mpathb (254:3) ├─ (8:192) ├─ (8:112) └─ (8:64) mpatha (254:2) ├─ (8:160) ├─ (8:96) └─ (8:32) vg00-lvswap (254:0) └─ (8:2) vg00-lvroot (254:1) ├─ (8:16) └─ (8:2)
（8）客户端 yast iscsi-client 工具查看
iscsi的其他常用操作（客户端）
（1）列出所有target
# iscsiadm -m node
（2）连接所有target
# iscsiadm -m node -l all
（3）连接指定target
# iscsiadm -m node -t iqn.... -p 172.29.88.62 --login
（4）使用如下命令可以查看配置信息
# iscsiadm -m node -o show -t iqn.2000-01.com.synology:rackstation.exservice-bak
（5）查看目前 iscsi target 连接状态
# iscsiadm -m session # iscsiadm: no active sessions.
(目前没有已连接的 iscsi target)
（6）断开所有target
# iscsiadm -m node -u all
（7）断开指定target
# iscsiadm -m node -t iqn... -p 172.29.88.62 --logout
（8）删除所有node信息
# iscsiadm -m node --op delete
（9）删除指定节点（/var/lib/iscsi/nodes目录下，先断开session）
# iscsiadm -m node -o delete -name iqn.2012-01.cn.nayun:test-01
（10）删除一个目标（/var/lib/iscsi/send_targets目录下）
# iscsiadm --mode discovery -o delete -p 172.29.88.62:3260
上一篇： Linux在丢失的情况下重置密码