欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

Azure Storage 分块上传

程序员文章站 2023-12-30 19:00:52
...

概述


Azure 存储提供三种类型的 Blob:块 Blob、页 Blob 和追加 Blob。其中,块 Blob 特别适用于存储短的文本或二进制文件,例如文档和媒体文件。

块 Blob 由块组成,每个块可以是不同的大小,最大为 100MB (对于 2016-05-31 之前 REST 版本的请求为 4MB ),块 Blob 最多可以包含 50,000 块。因此,块 Blob 的最大大小约为 4.75 TB (100MB X 50,000 块)。对于 2016-05-31 之前的 REST 版本,块 Blob 的最大大小约为 195 GB(4MB X 50,000 块),更多详细信息,请参阅块 Blob、追加 Blob 以及页 Blob 介绍


在上传文件到 Azure Blob 存储时,Azure 支持两种方式,整体上传和分块上传。

  • 整块上传:当上传到块 Blob 的文件小于等于 SingleBlobUploadThresholdInBytes 属性(客户端可以通过设置该属性设置单个 Blob 上传的最大值,范围介于 1MB 和 256MB 之间)的值时,则可以采用整体上传的方式,调用 PutBlob 完整的上传 Blob ,更多详细信息,请参考 PutBlob

  • 分块上传:当上传的块 Blob 的文件大于 SingleBlobUploadThresholdInBytes 属性的值时,存储客户端会根据 StreamWriteSizeInBytes (客户端可以通过设置该属性设置单个分块 Blob 的大小,范围介于 16KB 和 100MB 之间) 的值将文件分解成块, 采用分块上传的方式上传文件,更多详细信息,请参考 PubBlobList


关于C#的示例目前官网已经给出,下面分别给出JAVA、Python和PHP语言的示例程序:

  • JAVA Code Sample

import com.microsoft.azure.storage.CloudStorageAccount;
import com.microsoft.azure.storage.StorageException;
import com.microsoft.azure.storage.blob.BlockEntry;
import com.microsoft.azure.storage.blob.CloudBlobClient;
import com.microsoft.azure.storage.blob.CloudBlobContainer;
import com.microsoft.azure.storage.blob.CloudBlockBlob;
import java.io.FileInputStream;
import java.io.IOException;
import java.net.URISyntaxException;
import java.security.InvalidKeyException;
import java.util.ArrayList;
import java.util.Base64;

public class uploadFileBlocksAsBlockBlob {

    public static void main(String[] args) throws IOException, StorageException, URISyntaxException, InvalidKeyException {

        FileInputStream fileInputStream = null;
        //存储连接字符串
        String storageConnectionString = "<storage connection string>";
        //上传容器的名称
        String containerName= "aaa1";
        // Parse the connection string and create a blob client to interact with Blob storage
        CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
        CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
        CloudBlobContainer container = blobClient.getContainerReference(containerName);
        CloudBlockBlob blockBlob = container.getBlockBlobReference("testblockblob");//希望上传后文件的名称

        //本地文件的位置
        String filePath = "D:\\test.zip";

        try {
            // Open the file
            fileInputStream = new FileInputStream(filePath);
            // Split the file into 32K blocks (block size deliberately kept small for the demo) and upload all the blocks
            int blockNum = 0;
            String blockId = null;
            String blockIdEncoded = null;
            ArrayList<BlockEntry> blockList = new ArrayList<BlockEntry>();
            while (fileInputStream.available() > (32 * 1024)) {
                blockId = String.format("%05d", blockNum);
                blockIdEncoded = Base64.getEncoder().encodeToString(blockId.getBytes());
                blockBlob.uploadBlock(blockIdEncoded, fileInputStream, (32 * 1024));
                blockList.add(new BlockEntry(blockIdEncoded));
                blockNum++;
            }
            blockId = String.format("%05d", blockNum);
            blockIdEncoded = Base64.getEncoder().encodeToString(blockId.getBytes());
            blockBlob.uploadBlock(blockIdEncoded, fileInputStream, fileInputStream.available());
            blockList.add(new BlockEntry(blockIdEncoded));

            // Commit the blocks
            blockBlob.commitBlockList(blockList);
        }
        catch (Throwable t) {
            throw t;
        }
        finally {
            // Close the file output stream writer
            if (fileInputStream != null) {
                fileInputStream.close();
            }
        }
    }
}

  • Python Code Sample

from azure.storage.blob import BlockBlobService, PublicAccess
from azure.storage.blob.models import BlobBlock
from AzureStorageDemo.RandomData import RandomData

# Block Blob Operations
def block_blob_operations():
    # Define the upload file
    file_to_upload = "test.mp4"
    # Define the block size
    block_size = 1024*1024*4 #4MB

    # Create blockblob_service object
    blockblob_service = BlockBlobService(
        connection_string='<storage connection string>')

    # Define container name
    container_name = 'blockblobcontainer' + RandomData().get_random_name(6)
    print("Container name:" + container_name)

    try:
        # Create a new container
        print('1. Create a container with name - ' + container_name)
        blockblob_service.create_container(container_name)

        blocks = []

        # Read the file
        print('2. Upload file to block blob')
        with open(file_to_upload, "rb") as file:
            file_bytes = file.read(block_size)
            while len(file_bytes) > 0:
                block_id = RandomData().get_random_name(32)
                blockblob_service.put_block(container_name, file_to_upload, file_bytes, block_id)

                blocks.append(BlobBlock(id=block_id))

                file_bytes = file.read(block_size)

        blockblob_service.put_block_list(container_name, file_to_upload, blocks)

        print('3. Get the block list')
        blockslist = blockblob_service.get_block_list(container_name, file_to_upload, None, 'all')
        blocks = blockslist.committed_blocks

        print('4. Enumerate blocks in block blob')
        for block in blocks:
            print('Block ' + block.id)
    finally:
        print('5. Delete container')
        if blockblob_service.exists(container_name):
            blockblob_service.delete_container(container_name)

# Main method.
if __name__ == '__main__':
    block_blob_operations()

  • PHP Code Sample

<?php
require_once 'vendor\autoload.php';

use MicrosoftAzure\Storage\Common\ServicesBuilder;
use MicrosoftAzure\Storage\Common\ServiceException;
use MicrosoftAzure\Storage\Blob\Models\BlockList;

// Create blob REST proxy.
$connectionString = 'BlobEndpoint=https://<storage account name>.blob.core.chinacloudapi.cn/;QueueEndpoint=https://<storage account name>.queue.core.chinacloudapi.cn/;TableEndpoint=https://<storage account name>.table.core.chinacloudapi.cn/;AccountName=<storage account name>;AccountKey=<storage account key>';
$blobRestProxy = ServicesBuilder::getInstance()->createBlobService($connectionString);

$container = "aaaa";
$file_name = 'bigfile';
$blob_name = basename($file_name);
$block_list = new BlockList();
define('CHUNK_SIZE', 4 * 1024 * 1024);#设置每个块的大小为4MB
try {
    $fptr = fopen($file_name, "rb");
    $index = 1;
    while (!feof($fptr)) {
        $block_id = base64_encode(str_pad($index, 6, "0", STR_PAD_LEFT));
        $block_list->addUncommittedEntry($block_id);
        $data = fread($fptr, CHUNK_SIZE);
        $blobRestProxy->createBlobBlock($container, $blob_name, $block_id, $data);
        ++$index;
    }
    $blobRestProxy->commitBlobBlocks($container, $blob_name, $block_list);
} catch (ServiceException $e) {
    $code = $e->getCode();
    $error_message = $e->getMessage();
    echo $code.": ".$error_message."<br />";
}

echo 'upload big file success'
?>

参考链接:

上传大文件到 Azure 存储块 Blob

azure-storage-java

azure-storage-python

Uploading Large File By Splitting Into Blocks In Windows Azure Blob Storage Using Windows Azure SDK For PHP

上一篇:

下一篇: