MapReduce 动态设置 namenode HDFS 的URI 博客分类： HBasehadoopzookeeper

开发MapReduce程序 一直都是在namenode 那台pc上用eclipse 直接运行，没有任何问题

今天在一台namenode上用eclipse 编译一M/R小程序，结果死活 都是

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: Unknown protocol to job tracker: org.apache.hadoop.hdfs.protocol.ClientProtocol
	at org.apache.hadoop.mapred.JobTracker.getProtocolVersion(JobTracker.java:344)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1383)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1379)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1377)

	at org.apache.hadoop.ipc.Client.call(Client.java:1030)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
	at $Proxy1.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:364)
	at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:208)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:175)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1310)
	at org.apache.hadoop.fs.FileSystem.access$100(FileSystem.java:65)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1328)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:109)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:210)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:372)
	at gucas.xiaoxia.InvertedIndex.main(InvertedIndex.java:110)

 baidu google了半天 也没找出如何在MapReduce之前动态设置 DFS的URI 

无奈之下 只能求助与 M/R 源代码 发现 org.apache.hadoop.fs.FileSystem

java.lang.Object
  org.apache.hadoop.conf.Configured
      org.apache.hadoop.fs.FileSystem

 

setDefaultUri

public static void setDefaultUri(Configuration conf,
                                 String uri)
Set the default filesystem URI in a configuration.
Parameters:
conf - the configuration to alter
uri - the new default filesystem uri

 setDefaultUri 就是他了。

      Configuration conf = new Configuration();

		
		String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
		System.out.println(FileSystem.getDefaultUri(conf)); 
		FileSystem.setDefaultUri(conf, new URI("hdfs://gucas-s2:9000")); //动态设置 dfs uri
		System.out.println(FileSystem.getDefaultUri(conf));
		if(otherArgs.length != 2){
		    System.err.println("Usage: invertedindex <in> <out>");
		    System.exit(2);
		}
		
		Job job = new Job(conf, "InvertedIndex");
		job.setJarByClass(InvertedIndex.class);

 

 一定要在Job job = new Job(conf, "InvertedIndex"); 之前设置
 FileSystem.setDefaultUri(conf, new URI("hdfs://gucas-s2:9000")); //动态设置 dfs uri

否则无效