YARN REST API 提交MR任务
程序员文章站
2022-04-01 16:38:49
...
@羲凡——只为了更好的活着
YARN REST API 提交MR任务
官网介绍地址 http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
既然是api,推荐使用postman或者idea,本文使用postman
0.准备MR的jar包并上传到hdfs上
hdfs dfs -put xifan227-1.0-SNAPSHOT.jar /cpos/test/
1.生成新的applicationId
请求类型:POST
URL:http://rm-http-address:port/ws/v1/cluster/apps/new-application
2.查看MR的jar包信息
请求类型:GET
URL:http://nm-http-address:port/webhdfs/v1/cpos/test/xifan227-1.0-SNAPSHOT.jar?op=GETFILESTATUS
3.提交应用
Cluster Applications API(Submit Application)
请求类型:POST
URL:http://rm-http-address:port/ws/v1/cluster/apps
Body:
{
"application-id":"application_1586340793234_0125",
"application-name":"AaronTest",
"queue":"default",
"priority":-2,
"am-container-spec":
{
"local-resources":
{
"entry":
[
{
"key":"xifan227-1.0-SNAPSHOT",
"value":
{
"resource":"hdfs://test96.com:8020/cpos/test/xifan227-1.0-SNAPSHOT.jar",
"type":"FILE",
"visibility":"APPLICATION",
"size": 30301,
"timestamp": 1586400852288
}
}
]
},
"commands":
{
"command":"/opt/modules/hadoop-3.2.1/bin/yarn jar xifan227-1.0-SNAPSHOT mr.WordcountMapreduce 1><LOG_DIR>/WC.stdout 2><LOG_DIR>/WC.stderr"
}
},
"max-app-attempts":1,
"resource":{"memory":1024, "vCores":2},
"application-type":"MAPREDUCE"
}
4.结果展示
细心的朋友会发现:
我明明只提交了一个application(application-id为application_1586340793234_0125),但是在8088界面上却多出来一个application-id为application_1586340793234_0126,而且最蛋疼的是编号125的应用失败了,但是126的却成功了。这个问题特别尴尬,有解决的朋友指导一下吧。
若对博客中有任何问题,欢迎留言交流
@羲凡——只为了更好的活着
推荐阅读
-
spark利用yarn提交任务报:YARN application has exited unexpectedly with state UNDEFINED
-
spark利用yarn提交任务报:YARN application has exited unexpectedly with state UNDEFINED
-
hadoop-yarn集群中,利用hell脚本自动化提交spark任务
-
解析spark源码yarn-cluster模式任务提交
-
Hive提交到yarn的任务一直在running问题排查
-
记Spark提交任务到Yarn错误汇总
-
SparkLauncher提交jar任务到spark-yarn
-
spark-yarn提交任务
-
Spark yarn模式任务提交
-
idea以yarn-client 提交任务到yarn