欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

SpringBoot通过Minio实现大文件分片上传

程序员文章站 2024-02-19 10:57:40
...

最近开发功能需要大文件上传,正常会在业务代码中封装一层minio的上传,为了避免中转,实现了minio直传的方式。实现过程中也确实碰到了不少坑,只能查询文档,本着开箱即用,简单方便的原则,java版本的Minio sdk默认是不允许单独调用分片的相关方法,但是升级到8.0.3后可以通过继承MinioClient实现分片方法的使用,后来又碰到官方Minio版本的bug,提交issue,还好Minio github开发者处理非常迅捷,再次感谢,2021-02-04修复的,使用的开发者需要注意下。 大致描述下流程:

  1. 用户调用初始化接口,后端调用minio初始化,得到uploadId,生成每个分片的minio上传url
  2. 用户调用对应分片的上传地址,多次上传会覆盖
  3. 调用完成接口,后端查询所有上传的分片并合并
  • 自定义minioClient

public class CustomMinioClient extends MinioClient {

    protected CustomMinioClient(MinioClient client) {
        super(client);
    }

    public String initMultiPartUpload(String bucket, String region, String object, Multimap<String, String> headers, Multimap<String, String> extraQueryParams) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, ServerException, InternalException, XmlParserException, InvalidResponseException, ErrorResponseException {
        CreateMultipartUploadResponse response = this.createMultipartUpload(bucket, region, object, headers, extraQueryParams);

        return response.result().uploadId();
    }

    public ObjectWriteResponse mergeMultipartUpload(String bucketName, String region, String objectName, String uploadId, Part[] parts, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, ServerException, InternalException, XmlParserException, InvalidResponseException, ErrorResponseException {

        return this.completeMultipartUpload(bucketName, region, objectName, uploadId, parts, extraHeaders, extraQueryParams);
    }

    public ListPartsResponse listMultipart(String bucketName, String region, String objectName, Integer maxParts, Integer partNumberMarker, String uploadId, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws NoSuchAlgorithmException, InsufficientDataException, IOException, InvalidKeyException, ServerException, XmlParserException, ErrorResponseException, InternalException, InvalidResponseException {
        return this.listParts(bucketName, region, objectName, maxParts, partNumberMarker, uploadId, extraHeaders, extraQueryParams);
    }
}
  • 后端处理类

public Map<String, Object> initMultiPartUpload(String bucketName, String objectName, int totalPart) {
        Map<String, Object> result = new HashMap<>();
        try {
            String uploadId = customMinioClient.initMultiPartUpload(bucketName, null, objectName, null, null);

            result.put("uploadId", uploadId);
            List<String> partList = new ArrayList<>();

            Map<String, String> reqParams = new HashMap<>();
            //reqParams.put("response-content-type", "application/json");
            reqParams.put("uploadId", uploadId);
            for (int i = 1; i <= totalPart; i++) {
                reqParams.put("partNumber", String.valueOf(i));
                String uploadUrl = minioClient.getPresignedObjectUrl(
                        GetPresignedObjectUrlArgs.builder()
                                .method(Method.PUT)
                                .bucket(bucketName)
                                .object(objectName)
                                .expiry(1, TimeUnit.DAYS)
                                .extraQueryParams(reqParams)
                                .build());
                partList.add(uploadUrl);
            }
            result.put("uploadUrls", partList);
        } catch (Exception e) {
            logger.error("error: {}", e.getMessage(), e);
            return null;
        }

        return result;
    }

    public boolean mergeMultipartUpload(String bucketName, String objectName, String uploadId) {
        try {
            Part[] parts = new Part[1000];
            //此方法注意2020.02.04之前的minio服务端有bug
            ListPartsResponse partResult = customMinioClient.listMultipart(bucketName, null, objectName, 1000, 0, uploadId, null, null);
            int partNumber = 1;
            for (Part part : partResult.result().partList()) {
                parts[partNumber - 1] = new Part(partNumber, part.etag());
                partNumber++;
            }
            customMinioClient.mergeMultipartUpload(bucketName, null, objectName, uploadId, parts, null, null);
        } catch (Exception e) {
            logger.error("error: {}", e.getMessage(), e);
            return false;
        }

        return true;
    }

注意:此处做了很多简单的操作还需自行优化,比如:

  • 设定了最大仅允许1000个分片
  • 初始化生成上传签名地址header头相关操作

整个流程看起来很简单,开始做的时候查询资料网上没有这么操作的,再加上又碰到了官方小bug(bug:listParts),还好社区活跃,不然只能老实的在业务中自己实现分片了。