SpringBoot通过Minio实现大文件分片上传
程序员文章站
2024-02-19 10:57:40
...
最近开发功能需要大文件上传,正常会在业务代码中封装一层minio的上传,为了避免中转,实现了minio直传的方式。实现过程中也确实碰到了不少坑,只能查询文档,本着开箱即用,简单方便的原则,java版本的Minio sdk默认是不允许单独调用分片的相关方法,但是升级到8.0.3后可以通过继承MinioClient实现分片方法的使用,后来又碰到官方Minio版本的bug,提交issue,还好Minio github开发者处理非常迅捷,再次感谢,2021-02-04修复的,使用的开发者需要注意下。 大致描述下流程:
- 用户调用初始化接口,后端调用minio初始化,得到uploadId,生成每个分片的minio上传url
- 用户调用对应分片的上传地址,多次上传会覆盖
- 调用完成接口,后端查询所有上传的分片并合并
-
自定义minioClient
public class CustomMinioClient extends MinioClient {
protected CustomMinioClient(MinioClient client) {
super(client);
}
public String initMultiPartUpload(String bucket, String region, String object, Multimap<String, String> headers, Multimap<String, String> extraQueryParams) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, ServerException, InternalException, XmlParserException, InvalidResponseException, ErrorResponseException {
CreateMultipartUploadResponse response = this.createMultipartUpload(bucket, region, object, headers, extraQueryParams);
return response.result().uploadId();
}
public ObjectWriteResponse mergeMultipartUpload(String bucketName, String region, String objectName, String uploadId, Part[] parts, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, ServerException, InternalException, XmlParserException, InvalidResponseException, ErrorResponseException {
return this.completeMultipartUpload(bucketName, region, objectName, uploadId, parts, extraHeaders, extraQueryParams);
}
public ListPartsResponse listMultipart(String bucketName, String region, String objectName, Integer maxParts, Integer partNumberMarker, String uploadId, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws NoSuchAlgorithmException, InsufficientDataException, IOException, InvalidKeyException, ServerException, XmlParserException, ErrorResponseException, InternalException, InvalidResponseException {
return this.listParts(bucketName, region, objectName, maxParts, partNumberMarker, uploadId, extraHeaders, extraQueryParams);
}
}
-
后端处理类
public Map<String, Object> initMultiPartUpload(String bucketName, String objectName, int totalPart) {
Map<String, Object> result = new HashMap<>();
try {
String uploadId = customMinioClient.initMultiPartUpload(bucketName, null, objectName, null, null);
result.put("uploadId", uploadId);
List<String> partList = new ArrayList<>();
Map<String, String> reqParams = new HashMap<>();
//reqParams.put("response-content-type", "application/json");
reqParams.put("uploadId", uploadId);
for (int i = 1; i <= totalPart; i++) {
reqParams.put("partNumber", String.valueOf(i));
String uploadUrl = minioClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.PUT)
.bucket(bucketName)
.object(objectName)
.expiry(1, TimeUnit.DAYS)
.extraQueryParams(reqParams)
.build());
partList.add(uploadUrl);
}
result.put("uploadUrls", partList);
} catch (Exception e) {
logger.error("error: {}", e.getMessage(), e);
return null;
}
return result;
}
public boolean mergeMultipartUpload(String bucketName, String objectName, String uploadId) {
try {
Part[] parts = new Part[1000];
//此方法注意2020.02.04之前的minio服务端有bug
ListPartsResponse partResult = customMinioClient.listMultipart(bucketName, null, objectName, 1000, 0, uploadId, null, null);
int partNumber = 1;
for (Part part : partResult.result().partList()) {
parts[partNumber - 1] = new Part(partNumber, part.etag());
partNumber++;
}
customMinioClient.mergeMultipartUpload(bucketName, null, objectName, uploadId, parts, null, null);
} catch (Exception e) {
logger.error("error: {}", e.getMessage(), e);
return false;
}
return true;
}
注意:此处做了很多简单的操作还需自行优化,比如:
- 设定了最大仅允许1000个分片
- 初始化生成上传签名地址header头相关操作
- 等
整个流程看起来很简单,开始做的时候查询资料网上没有这么操作的,再加上又碰到了官方小bug(bug:listParts),还好社区活跃,不然只能老实的在业务中自己实现分片了。
上一篇: vue循环生成el-form-item时,绑定值的写法
下一篇: Ubuntu 筹建PHP环境