欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

hadoop部署的若干问题解决

程序员文章站 2022-06-26 08:47:59
hadoop部署的若干问题解决 1 清理系统内存cached方法,解决Out of memory问题 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-07-31 10:34:58,344 FATAL org... ......
hadoop部署的若干问题解决
1 清理系统内存cached方法,解决Out of memory问题

INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting

2018-07-31 10:34:58,344 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Thread.java:717)

at org.apache.hadoop.ipc.Server.start(Server.java:3071)

at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.serviceStart(ClientRMService.java:282)

测试了多种hadoop 关于内存配置xml的修改,均无效。

默认配置,可以排除hadoop配置问题,清理系统cached试试,果然有效。

清理系统cached方法

1.运行 sync 将 dirty 的内容写回硬盘

$sync

2.通过修改 proc 系统的 drop_caches 清理free的cache

# echo 3 > /proc/sys/vm/drop_caches

2 hadoop.tmp.dir配置

如果不修改,默认在/tmp/下面,可能会存在文件丢失。其他问题通过修改这个参数顺利启动。

core-site.xml

<property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hdtest/hadoop-3.0.3/tmp</value>
</property>

3 util.NativeCodeLoader问题未解决

2018-08-01 10:27:07,428 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2018-08-01 10:27:本地库没有加载,这种本地库加载目录

查了各种资料,试了一下各种修改Djava.library.path方法目前没有解决,初步排除没有找到路径的原因,怀疑是3.0.3自带的库有问题。

export HADOOP_HOME=/home/hadoop/hadoop-2.6.4

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"

4 虚拟内存超标2.6 GB of 2.1 GB virtual memory used. Killing container.

<!--jie解决这个问题

,501 INFO mapreduce.Job: Task Id : attempt_1533094470713_0005_m_000001_0, Status : FAILED

[2018-08-01 13:01:48.863]Container [pid=22796,containerID=container_1533094470713_0005_01_000003] is

running beyond virtual memory limits. Current usage: 606.3 MB of 1 GB physical memory used;

2.6 GB of 2.1 GB virtual memory used. Killing container.

--->

<property>

<name>yarn.nodemanager.vmem-check-enabled</name>

<value>false</value>

<description>Whether virtual memory limits will be enforced for containers</description>

</property>

<property>

<name>yarn.nodemanager.vmem-pmem-ratio</name>

<value>4</value>

<description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>

</property>

>yarn.nodemanager.vmem-check-enabled 默认true,会检查虚拟内存是否超标

yarn.nodemanager.vmem-pmem-ratio 虚拟内存可使用量物理内存倍数 ,默认2.1倍

5 waiting for AM container to be allocated

YarnApplicationState:

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.

2018-07-31 00:53:54,221 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532968834179_0001_000002 State change from SUBMITTED to SCHEDULED on event = ATTEMPT_ADDED

2018-07-31 00:56:54,299 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Error cleaning master

java.net.ConnectException: Call From cvm-dbsrv02/127.0.0.1 to cvm-dbsrv02:17909 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495)

at org.apache.hadoop.ipc.Client.call(Client.java:1437)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)

怀疑 路由端口没有开放。。。看rm日志 果然 17909端口没有开放。

iptables -I INPUT -p tcp --dport 0:65536 -s 10.16.xx -j ACCEPT

iptables -I INPUT -p tcp --dport 0:65536 -s 132.108.xx -j ACCEPT

iptables -I INPUT -p tcp --dport 0:65536 -s 127.0.0.1 -j ACCEPT

对本机开放所有端口

6 运行速度提高,cpu,内存占用提高方法

Hadoop默认使用 <memory:8192, vCores:8> ,top,free-g查看,运行最多占用10%cpu,内存还有大量空闲。如何提高运行资源占用,提高运行速度?

查看本机真实资源情况4路40核80线程,512G内存。

[root@cvm-dbsrv02 inputserv2675w]# cat /proc/cpuinfo| grep "physical id"| sort| uniq| wc -l

4

[root@cvm-dbsrv02 inputserv2675w]# cat /proc/cpuinfo| grep "cpu cores"| uniq

cpu cores : 10

[root@cvm-dbsrv02 inputserv2675w]# cat /proc/cpuinfo| grep "processor"| wc -l

80

[root@cvm-dbsrv02 inputserv2675w]#

4路40核80线程。

提供CPU使用率

yarn.xml

yarn.nodemanager.resource.cpu-vcores 默认8-->40 10%-预计提高到50%-》

yarn.nodemanager.resource.memory-mb 默认8192-->1024*40=40960

Amount of physical memory, in MB, that can be allocated for containers. If set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically calculated(in case of Windows and Linux). In other cases, the default is 8192MB.

Yarn.xml

<property>

<name>yarn.nodemanager.resource.cpu-vcores</name>

<value>40</value>

</property>

<property>

<name>yarn.nodemanager.resource.memory-mb</name>

<value>40960</value>

</property>

Cpu占用提高到了59%,预期之中。

8-08-01 13:58:29,121 INFO mapreduce.Job: The url to track the job: http://cvm-dbsrv02:8088/proxy/application_1533100404744_0003/

2018-08-01 13:58:29,122 INFO mapreduce.Job: Running job: job_1533100404744_0003

2018-08-01 13:58:36,256 INFO mapreduce.Job: Job job_1533100404744_0003 running in uber mode : false

2018-08-01 13:58:36,258 INFO mapreduce.Job: map 0% reduce 0%

2018-08-01 14:04:59,057 INFO mapreduce.Job: map 1% reduce 0%

2018-08-01 14:17:08,424 INFO mapreduce.Job: map 2% reduce 0%

^C[hdtest@cvm-dbsrv02 hadoop-3.0.3]$

[hdtest@cvm-dbsrv02 hadoop-3.0.3]$

任务执行效率明显提高

2018-08-01 14:31:40,369 INFO mapreduce.Job: Running job: job_1533105032226_0001

2018-08-01 14:31:49,535 INFO mapreduce.Job: Job job_1533105032226_0001 running in uber mode : false

2018-08-01 14:31:49,544 INFO mapreduce.Job: map 0% reduce 0%

2018-08-01 14:34:25,322 INFO mapreduce.Job: map 1% reduce 0%

2018-08-01 14:38:11,068 INFO mapreduce.Job: map 2% reduce 0%

2018-08-01 14:41:55,441 INFO mapreduce.Job: map 3% reduce 0%

2018-08-01 14:45:36,867 INFO mapreduce.Job: map 4% reduce 0%