欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

hadoop集群namenode同时挂datanode

程序员文章站 2022-04-23 19:40:01
背景:(测试环境)只有两台机器一台namenode一台namenode,但集群只有一个结点感觉不出来效果,在namenode上挂一个datanode就有两个节点,弊端见最后 操作非常简单(添加独立节点参照:http://www.cnblogs.com/pu20065226/p/8493316.htm ......

背景:(测试环境)只有两台机器一台namenode一台namenode,但集群只有一个结点感觉不出来效果,在namenode上挂一个datanode就有两个节点,弊端见最后

操作非常简单(添加独立节点参照:http://www.cnblogs.com/pu20065226/p/8493316.html)

1.修改namenode节点的slave文件,增加新节点信息

 

[hadoop@hadoop-master hadoop]$ pwd
/usr/hadoop/hadoop-2.7.5/etc/hadoop
[hadoop@hadoop-master hadoop]$ cat slaves
slave1
hadoop-master
[hadoop@hadoop-master hadoop]$ 

 

 

 

2.启动新datanodedatanodenodemanger进程

先确认namenode和当前的datanode中,etc/hoadoop/excludes文件中无待加入的主机,再进行下面操作

[hadoop@slave2 hadoop-2.7.5]$ sbin/hadoop-daemon.sh start datanode
starting datanode, logging to /usr/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-datanode-slave2.out
[hadoop@slave2 hadoop-2.7.5]$ sbin/yarn-daemon.sh start nodemanager
starting datanode, logging to /usr/hadoop/hadoop-2.7.5/logs/yarn-hadoop-datanode-slave2.out
[hadoop@slave2 hadoop-2.7.5]$
91284 SecondaryNameNode
90979 NameNode
91519 ResourceManager
41768 DataNode
41899 NodeManager
41999 Jps [hadoop@slave2 ~]$


3.在NameNode上刷新节点

 

[hadoop@hadoop-master ~]$ hdfs dfsadmin -refreshNodes
Refresh nodes successful
[hadoop@hadoop-master ~]$sbin/start-balancer.sh

4.在namenode查看当前集群情况,

确认节点已经正常加入

 

[hadoop@hadoop-master hadoop-2.7.5]$ hdfs dfsadmin -report
Configured Capacity: 58663657472 (54.63 GB)
Present Capacity: 35990061540 (33.52 GB)
DFS Remaining: 35989540864 (33.52 GB)
DFS Used: 520676 (508.47 KB)
DFS Used%: 0.00%
Under replicated blocks: 12
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (2):

Name: 192.168.48.129:50010 (hadoop-master)
Hostname: hadoop-master
Decommission Status : Normal
Configured Capacity: 38588669952 (35.94 GB)
DFS Used: 213476 (208.47 KB)
Non DFS Used: 16331292188 (15.21 GB)
DFS Remaining: 22257164288 (20.73 GB)
DFS Used%: 0.00%
DFS Remaining%: 57.68%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Mar 19 19:54:45 PDT 2018


Name: 192.168.48.132:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 20074987520 (18.70 GB)
DFS Used: 307200 (300 KB)
Non DFS Used: 6342303744 (5.91 GB)
DFS Remaining: 13732376576 (12.79 GB)
DFS Used%: 0.00%
DFS Remaining%: 68.41%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Mar 19 19:54:46 PDT 2018

网页查看

hadoop集群namenode同时挂datanode

 

 

 

 

 

 

 

弊端(来源网络):首先NameNode将文件命名空间的状态保存在状态中,比如哪个文件块在哪个datanode上,由于在较大的hadoop集群中,会存在很多文件块,这样就会占用NameNode很大的内存,所以不会浪费NameNode的计算资源 其次,对于长时间运行的集群来说,NameNode一致将命名空间的状态变化写入edits日志文件,时间久了该文件也会很大,只要将NameNode的存储规划的合理,是不会浪费存储的

hadoop集群重要的是保证namdenode的长期稳定运行,把datanode放在namenode上,增加了namenode的负担,datanode占用大量的磁盘io,网络流量可能导致hdfs响应慢,错误率增加,要进行大量错误恢复,这影响集群的稳定性。

至于namenode是否浪费资源,namenode要维护整个集群的(一,二级关系)一、目录树,文件元信息,二、块到数据节点的映射。对于一定规模的集群要消耗大量的内存,cpu资源。namenode还会把一级关系持久化到镜像文件中,并且用编辑日志保证数据被持久化。这也会占用大量的存储资源,同事,有大量的datanode节点,可能还有大量的客户端同namenode进行网络通信。综上,namenode资源并没浪费!