欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

编写Java代码对HDFS进行增删改查操作代码实例

程序员文章站 2023-12-15 17:55:40
本文实例为大家分享了java代码对hdfs进行增删改查操作的具体代码,供大家参考,具体内容如下 import java.io.file; import ja...

本文实例为大家分享了java代码对hdfs进行增删改查操作的具体代码,供大家参考,具体内容如下

import java.io.file;
import java.io.fileoutputstream;
import java.io.ioexception;
import java.net.uri;

import org.apache.commons.compress.utils.ioutils;
import org.apache.hadoop.conf.configuration;
import org.apache.hadoop.fs.blocklocation;
import org.apache.hadoop.fs.fsdatainputstream;
import org.apache.hadoop.fs.fsdataoutputstream;
import org.apache.hadoop.fs.filestatus;
import org.apache.hadoop.fs.filesystem;
import org.apache.hadoop.fs.path;

public class fileopreation {

	public static void main(string[] args) throws ioexception {
		//createfile();
		//deletefile();
		//copyfiletohdfs();
		//mkdirs();
		//deldirs();
		listdirectory();
		download();

	}
	public static void createfile() throws ioexception {
		string uri = "hdfs://alvis:9000";
		configuration configuration =new configuration();
		filesystem fsystem = filesystem.get(uri.create(uri), configuration);
		byte[] file_content_buff="hello hadoop world, test write file !\n".getbytes();
		path dfs = new path("/home/test.txt");
		fsdataoutputstream outputstream = fsystem.create(dfs);
		outputstream.write(file_content_buff.length);
	}
	public fileopreation() {
		// todo auto-generated constructor stub
	}public static void deletefile() throws ioexception {
		string uri = "hdfs://alvis:9000";
		configuration configuration =new configuration();
		filesystem fsystem = filesystem.get(uri.create(uri), configuration);
		path deletf = new path("/home/test.txt");
		boolean delresult = fsystem.delete(deletf,true);
		system.out.println(delresult==true?"删除成功":"删除失败");
	}
  
	public static void copyfiletohdfs() throws ioexception {
		string uri = "hdfs://alvis:9000";
		configuration configuration =new configuration();
		filesystem fsystem = filesystem.get(uri.create(uri), configuration);
		path src = new path("e:\\serializationtest\\apitest.txt");
		path dest_src = new path("/home");
		fsystem.copyfromlocalfile(src, dest_src);
	}
	
	public static void mkdirs() throws ioexception {
		string uri = "hdfs://alvis:9000";
		configuration configuration =new configuration();
		filesystem fsystem = filesystem.get(uri.create(uri), configuration);
		path src = new path("/test");
		fsystem.mkdirs(src);
		
	}
	
	public static void deldirs() throws ioexception {
		string uri = "hdfs://alvis:9000";
		configuration configuration = new configuration();
		filesystem fsystem = filesystem.get(uri.create(uri), configuration);
		path src = new path("/test");
		fsystem.delete(src);

	}
	
	public static void listdirectory() throws ioexception {
		string uri = "hdfs://alvis:9000";
		configuration configuration = new configuration();
		filesystem fsystem = filesystem.get(uri.create(uri), configuration);
		filestatus[] fstatus = fsystem.liststatus(new path("/output"));
		for(filestatus status : fstatus)
			if (status.isfile()) {
				system.out.println("文件路径:"+status.getpath().tostring());
				system.out.println("文件路径 getreplication:"+status.getreplication());
				system.out.println("文件路径 getblocksize:"+status.getblocksize());
				blocklocation[] blocklocations = fsystem.getfileblocklocations(status, 0, status.getblocksize());
				for(blocklocation location : blocklocations){
					system.out.println("主机名:"+location.gethosts()[0]);
					system.out.println("主机名:"+location.getnames()[0]);
			  }
		  }
			else {
				system.out.println("directory:"+status.getpath().tostring());
			}
	}
	
	public static void download() throws ioexception {
		configuration configuration = new configuration();
		configuration.set("fs.defaultfs", "hdfs://alvis:9000");
		filesystem fsystem =filesystem.get(configuration);
		fsdatainputstream inputstream =fsystem.open( new path("/input/wc.jar"));
		fileoutputstream outputstream = new fileoutputstream(new file("e:\\learnlife\\download\\wc.jar"));
		ioutils.copy(inputstream, outputstream);
		system.out.println("下载成功!");
	}
}

思想:

一、定义虚拟机接口

二、先拿到hdfs远程调用接口对象configuration

三、定义分布式文件系统filesystem对象获取对象

四、给定路径

五、用filesystem对象调用操作

以上所述是小编给大家介绍的java代码对hdfs进行增删改查操作详解整合,希望对大家有所帮助

上一篇:

下一篇: