添加链接
link管理
链接快照平台
  • 输入网页链接,自动生成快照
  • 标签化管理网页链接
问题:spark在yarn执行任务遇到如下的错误:
19/12/03 17:19:50 ERROR YarnScheduler: Lost executor 8 on hd060.corp.yodao.com: Container marked as failed: container_e84_1574035425015_31355_01_000009 on host: hd060.corp.yodao.com. Exit status: 134. Diagnostics: Exception from container-launch.
Container id: container_e84_1574035425015_31355_01_000009
Exit code: 134
Exception message: /bin/bash: line 1: 122080 Aborted                 /disk1/eadop/jdk1.8.0_77/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms10240m -Xmx10240m '-Dodis.home=/global/exec/zhanghl/corpus-spark-temp1' '-Djava.io.tmpdir=/disk2/zhanghl/tmp' -Djava.io.tmpdir=/disk1/eadop/hadoop-tmp/nm-local-dir/usercache/zhanghl/appcache/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/tmp '-Dspark.driver.port=22789' -Dspark.yarn.app.container.log.dir=/disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://[email protected]:22789 --executor-id 8 --hostname hd060.corp.yodao.com --cores 2 --app-id application_1574035425015_31355 --user-class-path file:/disk1/eadop/hadoop-tmp/nm-local-dir/usercache/zhanghl/appcache/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/__app__.jar > /disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/stdout 2> /disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/stderr
Stack trace: ExitCodeException exitCode=134: /bin/bash: line 1: 122080 Aborted                 /disk1/eadop/jdk1.8.0_77/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms10240m -Xmx10240m '-Dodis.home=/global/exec/zhanghl/corpus-spark-temp1' '-Djava.io.tmpdir=/disk2/zhanghl/tmp' -Djava.io.tmpdir=/disk1/eadop/hadoop-tmp/nm-local-dir/usercache/zhanghl/appcache/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/tmp '-Dspark.driver.port=22789' -Dspark.yarn.app.container.log.dir=/disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://[email protected]:22789 --executor-id 8 --hostname hd060.corp.yodao.com --cores 2 --app-id application_1574035425015_31355 --user-class-path file:/disk1/eadop/hadoop-tmp/nm-local-dir/usercache/zhanghl/appcache/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/__app__.jar > /disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/stdout 2> /disk7/eadop/hadoop-tmp/userlogs/application_1574035425015_31355/container_e84_1574035425015_31355_01_000009/stderr
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
	at org.apache.hadoop.util.Shell.run(Shell.java:456)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 134
1.在这段错误附近找到applicationId

类似这样的 id: application_1574035425015_31345

2.执行:

yarn logs -applicationId application_1574035425015_31345

搜索Error,你会在打印的日志中看到Error的原因。
或者spark提交任务遇到错误执行不成功,而又找不到错误,或者打印的日志中想输出错误的示例,但是打印不出来,也同样的,在命令行显示的日志或者自己的输出nohup.txt中找application先关的id,执行上诉命令 即可。

其他参考:
ExitCodeException exitCode=1
此篇博客最下面
Container exited with a non-zero exit code 134
yarn报错ExitCodeException exitCode=127

Hadoop解决yarn报错Container exited with a non-zero exit code 1. Error file: prelaunch.err.
1 错误代码 Container exited with a non-zero exit code 134. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err collect_set 、 collect_list 某个key值(null, unknown, 空串)过多, 被聚合的value值过多, 导致范围(数组超范围)超限; 3 解决方法 1) 调大内存
产生这个错误原因网上千奇百怪,后来我发现我是因为没有给pycharm分配gpu空间 在代码前面加上这个分配gpu内存的命名一般就可以解决 os.environ[“CUDA_VISIBLE_DEVICES”] = “PCI_BUS_ID” os.environ[“CUDA_VISIBLE_DEVICES”] = ‘0’ config = tf.ConfigProto() config.gpu_op...
配置Pycharm,解决终端可运行,Pycharm报错:Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library libcudnn....
比如在a.h中@interfaceclass1:NSObject@end那么在b.h中,就不能再@interfaceclass1:NSObject@end了 如果重复定义重名的类,就会出现该错误 转载于:https://www.cnblogs.com/ashamp/p/3619284.html...
Container killed on request. Exit code is 143 [2021-03-28 09:34:25.568]Container exited with a non-zero exit code 143 1.不靠谱的方式:多运行几遍代码。巧了就不会遇到这个问题了 2.靠谱的方式: 在yarn-site.xml中添加如下配置信息,然后重启yarn <property> <description>Whether virtual memory limits
hadoop安装成功以后,执行内置jar包统计单词数量的时候出现错误: 错误内容: [root@bigdata01 hadoop]# hadoop jar /opt/software/hadoop_back/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar wordcount /input /output 2021-05-20 09:15:41,737 INFO client.RMProxy: Connecting to ResourceM
hadoop 集群迁移 , apache原生版本迁移到cdh版本时 , hive任务执行报错: Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : 因为未改动脚本, 以为是集群问题,期间尝试了以下方法, 发现并无效果: #在mapred-site.xml文件中添加mapreduce
For more detailed output, check the application tracking page: http://hadoop102:8088/cluster/app/application_1590917926429_0002 Then click on links to logs of each attempt. . Failing the applicati (4)“\u10100”//合法,相当于‘\u1010’和字符串“0”。 (5)移位运算符可以用于long int char short byte。 (6)类的访问控制符可以是public或什么都不加。 (7)goto是保留字但不是关键字。then什么都不是。 请问这几个也是出错的原因之一吗? IDEA出现no scala sdk in module intellij idea Benita008: 姐妹子,感谢你!我说每次打开旧项目都得重新选一次SDK,原来在这儿设置表情包表情包 三个线程轮流执行顺序打印ABC,依次是ABCABCABC......(三种方式) 火从木来: 第一种无法正常运行,不信你们自己试试,因为在方法上解锁,你用notifyAll(),但是实际上调用的是this.notyfyAll(),所以错误。所以用错锁来进行唤醒了 java 出现unreachable statement异常 原因检查 bobo1325: 果然,我也是这么出错的 记录一次的spark-submit报错: scheduler.ShuffleMapStage: ShuffleMapStage 0 is now unavailable on executor IDEA中scala广播变量 asScala报错 使用Jsoup.clean消除不受信任的HTML (防止XSS攻击)