欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

分布式对象存储服务minio安装部署及使用

程序员文章站 2022-04-18 12:41:15
...

参考文章:

     java操作minio最佳实践

     Minio 文件服务(1)—— Minio部署使用及存储机制分析

     Minio的部署与使用

     https://github.com/minio/minio

     https://docs.minio.io/

1. 部署 

1.1 单节点

1.1.1 容器部署

docker pull minio/minio
 
#在Docker中运行Minio单点模式
docker run -p 9000:9000 -e MINIO_ACCESS_KEY=sunseaiot -e MINIO_SECRET_KEY=sunseaiot minio/minio server /data
 
#要创建具有永久存储的Minio容器,您需要将本地持久目录从主机操作系统映射到虚拟配置~/.minio 并导出/data目录
#建立外挂文件夹 /Users/hbl/dockersp/volume/minio/
docker run -p 9000:9000 -e MINIO_ACCESS_KEY=sunseaiot -e 
MINIO_SECRET_KEY=sunseaiot -v /Users/hbl/dockersp/volume/minio/data:/data -v 
/Users/hbl/dockersp/volume/minio/config:/root/.minio minio/minio server /dat

1.1.2 二进制部署(linux)--use

wget https://dl.minio.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data

# use
# 启动后会打印出AccessKey和SecretKey等信息
./minio server /data/minio_oss_srv   

# 后台运行
 nohup ./minio server /data/minio_oss_srv > /data/logs/minio/minio.log 2>&1 & 

# 自定义MINIO_ACCESS_KEY和MINIO_SECRET_KEY
[aaa@qq.com minio]# export MINIO_ACCESS_KEY=minio
[aaa@qq.com minio]# export MINIO_SECRET_KEY=miniostorage
[aaa@qq.com minio]# ./minio server /data/minio_oss_srv 

#自定义端口号
[aaa@qq.com minio]# ./minio server --address 101.121.115.159:9001 /data/minio_oss_srv

1.2 多节点

分布式搭建的流程和单节点基本一样,Minio服务基于命令行传入的参数自动切换成单机模式还是分布式模式。

分布式Minio单租户存在最少4个盘最多16个盘的限制(受限于纠删码)。这种限制确保了Minio的简洁,
同时仍拥有伸缩性。如果你需要搭建一个多租户环境,你可以轻松的使用编排工具(Kubernetes)来管理多个Minio实例。

1.2.1 纠删码 (多块硬盘 / 服务)

项目     参数
最大驱动器数量  16
最小驱动器数量 4
读仲裁   N / 2
写仲裁 N / 2+1

(多节点部署如果要使用容器需要用docker swarm 和K8s,文档中有介绍,本文重点不在此,因此我直接在4个服务器上安装了Minio,直接运行即可。Minio服务基于命令行传入的参数自动切换成单机模式还是分布式模式,启动一个分布式Minio实例,你只需要把硬盘位置做为参数传给minio server命令即可,然后,你需要在所有其它节点运行同样的命令。)

1.2.2 部署4主机,每机单块磁盘(drive)

export MINIO_ACCESS_KEY=123456
export MINIO_SECRET_KEY=123456
./minio server http://192.168.8.110/export1 \
               http://192.168.8.111/export2 \
               http://192.168.8.112/export3 \
               http://192.168.8.113/export4

1.2.3 部署4主机,每机2块磁盘(drives)

export MINIO_ACCESS_KEY=123456
export MINIO_SECRET_KEY=123456
./minio server http://192.168.8.110/export1 http://192.168.1.110/export2 \
               http://192.168.8.111/export1 http://192.168.1.111/export2 \          
               http://192.168.8.112/export1 http://192.168.1.112/export2 \
               http://192.168.8.113/export1 http://192.168.1.113/export2

1.3. 后台运行

由于不是用docker部署的,所以需要将进程加入后台运行。使用nohup指令。

export MINIO_ACCESS_KEY=SunseaIoT2018!
export MINIO_SECRET_KEY=SunseaIoT2018!
nohup ./minio server http://192.168.8.110/minio1 \
               http://192.168.8.111/minio2 \
               http://192.168.8.112/minio3 \
               http://192.168.8.113/minio4 >  out.file  2>&1  & 

3. 使用

部署好Minio服务后可以通过浏览器访问。输入设置好的用户名和密码即可进行操作。

分布式对象存储服务minio安装部署及使用

3.1 API文档地址

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide

3.2 Java SDK访问Minio服务

public class MinioUtil {
 	private static String minio_url;
 	private static String minio_name;
 	private static String minio_pass;
  	private static String minio_bucketName;
  
	/**
	 * 
	 * @Title: uploadImage
	 * @Description:上传图片
	 * @param inputStream
	 * @param suffix
	 * @return
	 * @throws Exception
	 */
	public static JSONObject uploadImage(InputStream inputStream, String suffix) throws Exception {
		return upload(inputStream, suffix, "image/jpeg");
	}
  
	/**
	 * @Title: uploadVideo
	 * @Description:上传视频
	 * @param inputStream
	 * @param suffix
	 * @return
	 * @throws Exception
	 */
	public static JSONObject uploadVideo(InputStream inputStream, String suffix) throws Exception {
		return upload(inputStream, suffix, "video/mp4");
	}
 
 	/**
	 * @Title: uploadVideo
	 * @Description:上传文件
	 * @param inputStream
	 * @param suffix
	 * @return
	 * @throws Exception
	 */
	public static JSONObject uploadFile(InputStream inputStream, String suffix) throws Exception {
		return upload(inputStream, suffix, "application/octet-stream");
	}
  
	/**
	 * 上传字符串大文本内容
	 * 
	 * @Title: uploadString
	 * @Description:描述方法
	 * @param str
	 * @return
	 * @throws Exception
	 */
	public static JSONObject uploadString(String str) throws Exception {
		if (!StringUtils.notNullAndEmpty(str)) {
			return new JSONObject();
		}
		InputStream inputStream = new ByteArrayInputStream(str.getBytes());
		return upload(inputStream, null, "text/html");
	}
  
	/**
	 * @Title: upload
	 * @Description:上传主功能
	 * @return
	 * @throws Exception
	 */
	private static JSONObject upload(InputStream inputStream, String suffix, String contentType) 
             throws Exception {
		JSONObject map = new JSONObject();
		PropertiesLoader p = new PropertiesLoader("system.properties");
		minio_url = p.getProperty("minio_url");
		minio_name = p.getProperty("minio_name");
		minio_pass = p.getProperty("minio_pass");
		minio_bucketName = p.getProperty("minio_bucketName");
		MinioClient minioClient = new MinioClient(minio_url, minio_name, minio_pass);
		SimpleDateFormat sdf = new SimpleDateFormat("yyyyMM");
		String ymd = sdf.format(new Date());
		String objectName = ymd + "/" + UUID.randomUUID().toString() + (suffix != null ? suffix : "");
		minioClient.putObject(minio_bucketName, objectName, inputStream, contentType);
		String url = minioClient.getObjectUrl(minio_bucketName, objectName);
		map.put("flag", "0");
		map.put("mess", "上传成功");
		map.put("url", url);
		map.put("urlval", url);
		map.put("path", minio_bucketName + "/" + objectName);
		return map;
	}
}

3.3 Java SDK上传文件到Minio

package com.minio.client;
 
import io.minio.MinioClient;
import io.minio.errors.MinioException;
import lombok.extern.slf4j.Slf4j;
import org.xmlpull.v1.XmlPullParserException;
 
import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
 
@Slf4j
public class FileUploader {
    public static void main(String[] args) 
            throws NoSuchAlgorithmException, IOException, 
                   InvalidKeyException, XmlPullParserException {
        try {
            MinioClient minioClient =
                   new MinioClient("https://minio.sunseaiot.com", "sunseaiot", "sunseaiot",true);
 
            // 检查存储桶是否已经存在
            if(minioClient.bucketExists("ota")) {
                log.info("Bucket already exists.");
            } else {
                // 创建一个名为ota的存储桶
                minioClient.makeBucket("ota");
                log.info("create a new bucket.");
            }
 
            //获取下载文件的url,直接点击该url即可在浏览器中下载文件
            String url = minioClient.presignedGetObject("ota","hello.txt");
            log.info(url);
 
            //获取上传文件的url,这个url可以用Postman工具测试,在body里放入需要上传的文件即可
            String url2 = minioClient.presignedPutObject("ota","ubuntu.tar");
            log.info(url2);
 
            // 下载文件到本地
            minioClient.getObject("ota","hello.txt", "/Users/hbl/hello2.txt");
            log.info("get");
 
            // 使用putObject上传一个本地文件到存储桶中。
            minioClient.putObject("ota","tenant2/hello.txt", "/Users/hbl/hello.txt");
            log.info("/Users/hbl/hello.txt is successfully uploaded as hello.txt to `task1` bucket.");
        } catch(MinioException e) {
            log.error("Error occurred: " + e);
        }
    }
}

4. 启动服务文档

NAME:
  minio server - start object storage server

USAGE:
  minio server [FLAGS] DIR1 [DIR2..]
  minio server [FLAGS] DIR{1...64}

DIR:
  DIR points to a directory on a filesystem. When you want to combine
  multiple drives into a single large system, pass one directory per
  filesystem separated by space. You may also use a '...' convention
  to abbreviate the directory arguments. Remote directories in a
  distributed setup are encoded as HTTP(s) URIs.

FLAGS:
  --address value               bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname (default: ":9000")
  --config-dir value, -C value  [DEPRECATED] Path to legacy configuration directory. (default: "/root/.minio")
  --certs-dir value, -S value   Path to certs directory. (default: "/root/.minio/certs")
  --quiet                       Disable startup information.
  --anonymous                   Hide sensitive information from logging.
  --json                        Output server logs and startup information in json format.
  --help, -h                    Show help.
  
ENVIRONMENT VARIABLES:
  ACCESS:
     MINIO_ACCESS_KEY: Custom username or access key of minimum 3 characters in length.
     MINIO_SECRET_KEY: Custom password or secret key of minimum 8 characters in length.

  BROWSER:
     MINIO_BROWSER: To disable web browser access, set this value to "off".

  CACHE:
     MINIO_CACHE_DRIVES: List of mounted drives or directories delimited by ";".
     MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";".
     MINIO_CACHE_EXPIRY: Cache expiry duration in days.
     MINIO_CACHE_MAXUSE: Maximum permitted usage of the cache in percentage (0-100).

  DOMAIN:
     MINIO_DOMAIN: To enable virtual-host-style requests, set this value to Minio host domain name.

  WORM:
     MINIO_WORM: To turn on Write-Once-Read-Many in server, set this value to "on".

  BUCKET-DNS:
     MINIO_DOMAIN:    To enable bucket DNS requests, set this value to Minio host domain name.
     MINIO_PUBLIC_IPS: To enable bucket DNS requests, set this value to list of Minio host public IP(s) delimited by ",".
     MINIO_ETCD_ENDPOINTS: To enable bucket DNS requests, set this value to list of etcd endpoints delimited by ",".

   KMS:
     MINIO_SSE_VAULT_ENDPOINT: To enable Vault as KMS,set this value to Vault endpoint.
     MINIO_SSE_VAULT_APPROLE_ID: To enable Vault as KMS,set this value to Vault AppRole ID.
     MINIO_SSE_VAULT_APPROLE_SECRET: To enable Vault as KMS,set this value to Vault AppRole Secret ID.
     MINIO_SSE_VAULT_KEY_NAME: To enable Vault as KMS,set this value to Vault encryption key-ring name.

EXAMPLES:
  1. Start minio server on "/home/shared" directory.
     $ minio server /home/shared

  2. Start minio server bound to a specific ADDRESS:PORT.
     $ minio server --address 192.168.1.101:9000 /home/shared

  3. Start minio server and enable virtual-host-style requests.
     $ export MINIO_DOMAIN=mydomain.com
     $ minio server --address mydomain.com:9000 /mnt/export

  4. Start erasure coded minio server on a node with 64 drives.
     $ minio server /mnt/export{1...64}

  5. Start distributed minio server on an 32 node setup with 32 drives each. Run following command on all the 32 nodes.
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...32}.example.com/mnt/export/{1...32}

  6. Start minio server with edge caching enabled.
     $ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/drive3;/mnt/drive4"
     $ export MINIO_CACHE_EXCLUDE="bucket1/*;*.png"
     $ export MINIO_CACHE_EXPIRY=40
     $ export MINIO_CACHE_MAXUSE=80
     $ minio server /home/shared

  7. Start minio server with KMS enabled.
     $ export MINIO_SSE_VAULT_APPROLE_ID=9b56cc08-8258-45d5-24a3-679876769126
     $ export MINIO_SSE_VAULT_APPROLE_SECRET=4e30c52f-13e4-a6f5-0763-d50e8cb4321f
     $ export MINIO_SSE_VAULT_ENDPOINT=https://vault-endpoint-ip:8200
     $ export MINIO_SSE_VAULT_KEY_NAME=my-minio-key
     $ minio server /home/shared