欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  科技

Hadoop系列修炼笔记12

程序员文章站 2022-07-08 12:03:11
核心内容:压缩和解压缩的代码 package com.shujuelin.hdfs.mr.compress; import org.apache.hadoop.conf.Confi...

核心内容:压缩和解压缩的代码

package com.shujuelin.hdfs.mr.compress;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.compress.*;
import org.apache.hadoop.util.ReflectionUtils;
import org.junit.Test;

import java.io.FileInputStream;
import java.io.FileOutputStream;

/**
 * 测试压缩
 */
public class TextCompress {

    public static void main(String[] args) throws Exception {

        Class[] zipClasses = {
                DeflateCodec.class,
                GzipCodec.class,
                BZip2Codec.class,
                Lz4Codec.class,
                SnappyCodec.class
        };

        for(Class c : zipClasses){
            //unzip(c);
            zip(c);
        }
    }

    /**
     * 压缩测试
     */
    public static void zip(Class codecClass) throws Exception {
        long start = System.currentTimeMillis();
        //实例化对象CompressionCodec:压缩编解码器
        CompressionCodec codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, new Configuration());
        //创建文件输出流,得到默认扩展名
        FileOutputStream fos = new FileOutputStream("/home/centos/zip/b" + codec.getDefaultExtension());
        //得到压缩流
        CompressionOutputStream zipOut = codec.createOutputStream(fos);
        IOUtils.copyBytes(new FileInputStream("/home/centos/zip/hadoop.txt"), zipOut, 1024);
        zipOut.close();
        System.out.println(codecClass.getSimpleName() + " : " + (System.currentTimeMillis() - start) );
    }

    /**
     * 解压缩
     */
    public static void unzip(Class codecClass) throws Exception {
        long start = System.currentTimeMillis();
        //实例化对象
        CompressionCodec codec = (CompressionCodec)ReflectionUtils.newInstance(codecClass, new Configuration());
        //创建文件输出流,得到默认扩展名
        FileInputStream fis = new FileInputStream("/home/centos/zip/b" + codec.getDefaultExtension());
        //得到压缩流
        CompressionInputStream zipIn = codec.createInputStream(fis);
        IOUtils.copyBytes(zipIn,new FileOutputStream("/home/centos/zip/hadoop" + codec.getDefaultExtension() + ".txt"), 1024);
        zipIn.close();
        System.out.println(codecClass.getSimpleName() + " : " + (System.currentTimeMillis() - start) );
    }
}