hadoop自学记录（一）

程序员文章站 2024-03-02 16:08:58

...

【此篇纯属自己的笔记记录，无实际帮助价值】
强烈建议参考官方文档http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html 来安装hadoop。搜索引擎搜出来的很多内容，都是找不到内容的，比如hadoop解压后的目录下根本没有conf文件夹了。
刚开始接触hadoop不久，又在另外一台电脑上搭建一次hadoop环境。
按照http://www.cnblogs.com/kinglau/p/3794433.html来处理（下面这几个步骤参考这个网页的）
1.创建hadoop用户和组sudo addgroup hadoop
2.创建hadoop用户 sudo adduser -ingroup hadoop hadoop
3.为hadoop用户添加权限sudo vim /etc/sudoers

# User privilege specification
root    ALL=(ALL:ALL) ALL
hadoop  ALL=(ALL:ALL) ALL

4.安装ssh sudo apt-get install ssh
安装pdsh sudo apt-get install pdsh
5.查看服务是否启动 ps -e | grep ssh
6.设置免密码登录，生成私钥和公钥ssh-****** -t rsa -P ""
7.将公钥追加到authorized_keys中，它用户保存所有允许以当前用户身份登录到ssh客户端用户的公钥内容。cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
8.到了后面，发现是使用hadoop重新登录系统，然后再次安装jdk等环境。由于我目前的用户不是root，已经安装了jdk等环境，就不想按照其步骤继续。
直接将解压之后的hadoop文件夹移动到/usr/lib/目录下，然后修改~/.bashrc文件内容。最后使用WordCount示例。

HP-Pavilion-g4-Notebook-PC:/usr/lib/hadoop-2.8.0$ sudo mv ~/Downloads/hadoop-2.8.0/ /usr/lib/
HP-Pavilion-g4-Notebook-PC:/usr/lib/hadoop-2.8.0$ chmod 775 /usr/lib/hadoop-2.8.0/
HP-Pavilion-g4-Notebook-PC:/usr/lib/hadoop-2.8.0$ vim ~/.bashrc
HP-Pavilion-g4-Notebook-PC:/usr/lib/hadoop-2.8.0$ source ~/.bashrc
HP-Pavilion-g4-Notebook-PC:/usr/lib/hadoop-2.8.0$ bin/hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.0-sources.jar org.apache.hadoop.examples.WordCount ~/Documents/feature/negative/negative_Special_DR_Tag_result.txt output
HP-Pavilion-g4-Notebook-PC:/usr/lib/hadoop-2.8.0$ cat output/*

hadoop自学记录（一）

~/.bashrc内容

export JAVA_HOME=/usr/lib/jvm/java
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

export SCALA_HOME=/usr/lib/scala-2.12.1
export PATH=$PATH:${SCALA_HOME}/bin

export SPARK_HOME=/usr/lib/spark-2.1.0-bin-hadoop2.7
export PATH=$PATH:${SPARK_HOME}/bin

export MAVEN_HOME=/usr/lib/apache-maven-3.3.9
export PATH=$PATH:${MAVEN_HOME}/bin

#set weka enviroment
export WEKA_HOME=/usr/weka
export CLASSPATH=$CLASSPATH:$WEKA_HOME/weka.jar

#set hadoop enviroment
export HADOOP_HOME=/usr/lib/hadoop-2.8.0
export path=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

vim /usr/lib/hadoop-2.8.0/etc/hadoop/hadoop-env.sh

# set to the root of your Java installation
export JAVA_HOME=/usr/lib/jvm/java

注意修改这个JAVA_HOME路径，否则在后面使用时候，会报错

HP-Pavilion-g4-Notebook-PC:/usr/lib/hadoop-2.8.0$ sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 7f:77:9e:35:fe:21:22:6f:dd:4c:20:27:16:d1:43:37.
Are you sure you want to continue connecting (yes/no)?

参考网页
1. http://www.cnblogs.com/kinglau/p/3794433.html
2. 官方文档http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

上一篇： Vue实现通过数据找到对应的DOM节点

下一篇： Apache Ignite（六）：Ignite的集群部署

hadoop自学记录（一）