Solved: Container exited with a non-zero exit code 13. Err... - Cloudera Community

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

聪明的海龟 · 在线，菜鸟遇MyBatis ...· 2 月前 ·

发怒的弓箭 · 验证码· 5 月前 ·

英勇无比的大脸猫 · 留学干货 | 英国常用的租房买房网站推荐 ...· 6 月前 ·

玩命的松树 · Spring 事务执行流程及源码分析 - ...· 1 年前 ·

英俊的野马 · Redirect | ...· 1 年前 ·

Hello all,

I am trying to run spark job using spark-submit with a docker image over yarn.

I followed the instructions in the Blog provided by cloudera in the following link:

https://blog.cloudera.com/introducing-apache-spark-on-docker-on-top-of-apache-yarn-with-cdp-datacent...

and I ran into an error that I couldn't fine an answer to.

Note: I already did all the configurations required from the post.

I ran this command :

spark-submit \
--master yarn \
--deploy-mode cluster \
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=faresdev8/python3:v5 \
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data1/opt/cloudera/parcels/:/data1/opt/cloudera/parcels/:ro" \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=faresdev8/python3:v5 \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data1/opt/cloudera/parcels/:/data1/opt/cloudera/parcels/:ro" \
ols.py

And this is the error I get:

Sometimes its gives me an exit code 29. I don't understand what the problem is especially that I followed instructions properly.

Okay so I solved this problem.

if anyone got something related check on these 3 things:

1- spark version does not mismatch the python version. Spark 2 doesn't support python higher than 3.7.

2- Make sure that your python code starts a spark session, I forgot that I removed that when I was experimenting.

3- Make sure there are no problems in the code it self and test it on another machine to check if it works properly

Okay so I solved this problem.

if anyone got something related check on these 3 things:

1- spark version does not mismatch the python version. Spark 2 doesn't support python higher than 3.7.

2- Make sure that your python code starts a spark session, I forgot that I removed that when I was experimenting.

3- Make sure there are no problems in the code it self and test it on another machine to check if it works properly

Terms & Conditions

Unsubscribe / Do Not Sell My Personal Information

Supported Browsers Policy

Apache Hadoop and associated open source project names are trademarks of the Apache Software Foundation. For a complete list of trademarks, click here.