IDEA 下使用maven 创建 MapReduce(配置环境 Mac OS)
第一次使用 maven 创建项目,第一次碰到的坑不少,但是 maven 的确好用啊!!看别的好多博客写的教程不清楚,并且好多都是复制的,期间遇到了好多坑也没解释。简单说下步骤,和我遇到的坑~~~~
环境:IDEA JDK1.8(已配置)maven(官网下载的没使用 IDEA 自带的)
第一步:maven 搭建
1. 官网下载 http://maven.apache.org/download.cgi (mac 我下载的是)
2. 下载后解压 我解压到了 /Library/apache-maven
3. 输入命令配置环境变量 vi ~/.bash_profile
4. export M2_HOME=/Library/apache-maven
5. PATH=”$M2_HOME/bin”自己之前配的别的不要删除
6. 配置文件生效 source ~/.bash_profile
7.查看 maven 输入命令 mvn -version查看是不是安装成功
第二步:配置 IDEA 下 maven 的设置
1. 在 maven 中配置这三个
2. maven home directory :自己安装 maven 的目录
3. User settings file:这个就是在 maven 目录下的/conf 下的配置文件(很重要,下面要配置)
4. Local repository:这个是下载的 jar 包的目录(没有这个 repository 文件,自己创建一个)
5. 默认(3和4文件)文件在 用户名下的 ~/.m2文件里面(自己配置了好管理)
第三步:配置 settings 文件
1.这个就是 jar 包下载目录的配置
2.使用*仓库,不然下载 jar 包太慢了
3.传一下代码,自己也可以配置别的*仓库。
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
第四步:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.hubo</groupId>
<artifactId>bigdata</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId></artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.9.0</version>
</dependency>
</dependencies>
</project>
配置之后就会导入很多包
第四步:写程序
把之前写的程序放进去
package MapReduce;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
/**
* Created by hubo on 2017/12/3
*/
public class wordcount {
private static final String OUT = "hdfs://hdp01:9000/wordcount/output/";
public static class Map extends Mapper<LongWritable,Text,Text,LongWritable>{
@Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String[] words = value.toString().split(" ");
for(String word : words){
context.write(new Text(word),new LongWritable(1L));
}
}
}
public static class Reduce extends Reducer<Text,LongWritable,Text,LongWritable>{
@Override
protected void reduce(Text key, Iterable<LongWritable> values, Context context)
throws IOException, InterruptedException {
Long count = 0L;
for(LongWritable value : values){
count += value.get();
}
context.write(key,new LongWritable(count));
}
}
public static void main(String[] args) throws Exception{
//设置环境变量HADOOP_USER_NAME,其值是root
//在本机调试
System.setProperty("HADOOP_USER_NAME", "root");
//读取配置文件
Configuration conf = new Configuration();
conf.set("fs.defaultFS","hdfs://192.168.56.101:9000");
conf.set("yarn.resourcemanager.hostname","hdp01");
FileSystem fs = FileSystem.get(conf);
Job job = Job.getInstance(conf,"Demo");
job.setJarByClass(wordcount.class); //主类
job.setMapperClass(Map.class);
//combine过程发生在map方法和reduce方法之间,它将中间结果进行了一次合并。
job.setCombinerClass(Reduce.class);
job.setReducerClass(Reduce.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
job.setNumReduceTasks(2);
Path out = new Path(OUT);
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job,new Path("hdfs://hdp01:9000/wordcount/input/"));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job,out);
if(fs.exists(out)){
fs.delete(out, true);
}
System.exit(job.waitForCompletion(true) ? 0:1 );
}
}
在IDEA中运行MapReduce程序,参数设置都正确的情况下,运行时控制台的输出为:
log4j:WARN No appenders could be found for logger (org.apache.Hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
解决方法:将 hadoop 下的/etc/hadoop/里面的 log4.properties 拷贝到
项目下面的
之后运行就可以了