[转] lzo本地压缩与解压缩实例
程序员文章站
2022-06-13 23:24:53
...
- /**
- * @author HJX
- * @version 1.0,2013-01-16
- * @since JDK1.7,Ubuntu-12.04-64bit
- * 在hadoop环境下运行
- * 将一个String写入到本地lzo文件中(不是hadoop的hdfs上)
- * 再从该lzo文件中读取出来并与原String进行校对
- */
- import java.io.BufferedReader;
- import java.io.FileInputStream;
- import java.io.FileNotFoundException;
- import java.io.FileOutputStream;
- import java.io.IOException;
- import java.io.InputStream;
- import java.io.InputStreamReader;
- import java.io.OutputStream;
- import java.util.ArrayList;
- import java.util.List;
- import org.apache.hadoop.conf.Configuration;
- import com.hadoop.compression.lzo.LzopCodec;
- public class LzoCompress {
- /**
- * @param args
- */
- public static void main(String[] args) {
- //生成数据
- String dataSource = "abcdefghijklmnopqrstuvwxyz0123456789~!%#^@*#*%$(\n";
- dataSource = dataSource.concat(dataSource);
- dataSource = dataSource.concat(dataSource);
- dataSource = dataSource.concat(dataSource);
- /* System.out.println("dataSource = " + dataSource);*/
- String lzoFilePath = "/home/hadoop/LzoCompressTest.lzo";
- //写入到lzo文件,即lzo压缩
- write2LzoFile(lzoFilePath, getDefaultConf(),dataSource.getBytes());
- StringBuilder sb = new StringBuilder();
- //读取lzo文件,即lzo解压缩
- List<String> lines = readLzoFile(lzoFilePath, getDefaultConf());
- for(String line : lines) {
- sb.append(line);
- //LINUX/UNIX 下添加一个换行符
- sb.append("\n");
- /* //Windows 下添加一个换行符
- sb.append("\r\n");*/
- }
- if (sb.toString().equals(dataSource)) {
- System.out.println(sb.toString());
- } else {
- System.err.println("Error line : " + sb.toString());
- }
- }
- private static Configuration getDefaultConf(){
- Configuration conf = new Configuration();
- conf.set("mapred.job.tracker", "local");
- conf.set("fs.default.name", "file:///");
- conf.set("io.compression.codecs", "com.hadoop.compression.lzo.LzoCodec");
- return conf;
- }
- /**
- * 写数据到lzo文件,即lzo压缩
- * @param destLzoFilePath
- * @param conf
- * @param datas
- * @return void
- */
- public static void write2LzoFile(String destLzoFilePath,Configuration conf,byte[] datas) {
- LzopCodec lzo = null;
- OutputStream out = null;
- try {
- /* System.setProperty("java.library.path", "/usr/local/hadoop/lib/native/Linux-amd64-64/lib");*/
- lzo = new LzopCodec();
- lzo.setConf(conf);
- out = lzo.createOutputStream(new FileOutputStream(destLzoFilePath));
- out.write(datas);
- } catch (FileNotFoundException e) {
- e.printStackTrace();
- } catch (IOException e) {
- e.printStackTrace();
- } finally {
- try {
- if(out != null) {
- out.close();
- }
- } catch (IOException e) {
- e.printStackTrace();
- }
- }
- }
- /**
- * 从lzo文件中读取数据,即lzo解压缩
- * @param lzoFilePath
- * @param conf
- * @return void
- */
- public static List<String> readLzoFile(String lzoFilePath,Configuration conf) {
- LzopCodec lzo = null;
- InputStream is = null;
- InputStreamReader isr = null;
- BufferedReader reader = null;
- List<String> result = null;
- String line = null;
- try {
- /* System.setProperty("java.library.path", "/usr/local/hadoop/lib/native/Linux-amd64-64/lib");*/
- lzo = new LzopCodec();
- lzo.setConf(conf);
- is = lzo.createInputStream(new FileInputStream(lzoFilePath));
- isr = new InputStreamReader(is);
- reader = new BufferedReader(isr);
- result = new ArrayList<String>();
- while((line = reader.readLine()) != null) {
- result.add(line);
- }
- } catch (FileNotFoundException e) {
- e.printStackTrace();
- } catch (IOException e) {
- e.printStackTrace();
- } finally {
- try {
- if (reader != null) {
- reader.close();
- }
- if (isr != null) {
- isr.close();
- }
- if (is != null) {
- is.close();
- }
- } catch (IOException e) {
- e.printStackTrace();
- }
- }
- return result;
- }
- }
程序是没有错的,但是一开始运行的时候总会提示无法读取libgplcompression这个库,其实我知道少了哪些库的,分别是
libgplcompression.a
libgplcompression.la
libgplcompression.so
libgplcompression.so.0
libgplcompression.so.0.0.0
可问题是把这些库放在哪里。尝试过把这几个库放在$CLASSPATH下面,但没用。于是查看了错误提示,提示缺少的这个库在 com.hadoop.compression.lzo.GPLNativeCodeLoader这个类里面被引用到,于是看了一下hadoop- lzo-0.45.jar的源文件(当时编译hadoop-lzo-0.45.jar时留下的源文件,在kevinweil-hadoop-lzo- 6bb1b7f/src/java/com/hadoop/compression/lzo/里),GPLNativeCodeLoader.java的 内容是这样的:
- package com.hadoop.compression.lzo;
- import org.apache.commons.logging.Log;
- import org.apache.commons.logging.LogFactory;
- public class GPLNativeCodeLoader {
- private static final Log LOG = LogFactory.getLog(GPLNativeCodeLoader.class);
- private static boolean nativeLibraryLoaded = false;
- static {
- try {
- //try to load the lib
- System.loadLibrary("gplcompression");
- nativeLibraryLoaded = true;
- LOG.info("Loaded native gpl library");
- } catch (Throwable t) {
- LOG.error("Could not load native gpl library", t);
- nativeLibraryLoaded = false;
- }
- }
- /**
- * Are the native gpl libraries loaded?
- * @return true if loaded, otherwise false
- */
- public static boolean isNativeCodeLoaded() {
- return nativeLibraryLoaded;
- }
- }
这里跟load那个libgplcompression库 有关的语句应该是try语句块里面的那个System.loadLibrary("gplcompression");
于是我再查了一下这个loadLibrary的动作到底是怎样的动作。于是在这篇blog里找到了解答:http://blog.csdn.net/forandever/article/details/5983846
System.loadLibrary()load的是 java.library.path这一jvm变量所指向的路径中的库。那我只要把那些libgplcompression库 所在的文件夹加入到java.library.path里面不就行了~于是我查找设置java.library.path的方法,
方法1:命令行
java -Djava.library.path=/path/to/libgplcompression/ ***.class
方法2:java语句
在程序里加入这么一句,System.setProperty("java.library.path", "/path/to/libgplcompression/");
就找了这么2个方法,可是这2个方法都只能临时改变java.library.path的值!
除了这2个方法,我找不到别的方法了,累死了,不再找了,索性把libgplcompression这些库给copy到java.library.path指向的文件夹里!
copy完后,再次执行,OK了!
要获取java.library.path的值,可以用java语句
System.out.println(System.getProperty("java.library.path"));
我的是
/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
参考资料:
gpllibcompression库以及hadoop-lzo-0.4.15.jar下载链接
上一篇: web - php 实现同一条链接点击只显示一次内容
下一篇: 安装laravel