Solved: Re: When running Spark job getting Error: java.lan... - Cloudera Community

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

豪情万千的小刀 · itext 两个pdf合并的时候 ...· 3 周前 ·

欢乐的柳树 · 上海植物园· 2 月前 ·

重感情的手术刀 · 仿element-ui ...· 2 月前 ·

朝气蓬勃的油条 · [Bug?]: DateTime in ...· 2 月前 ·

可爱的伏特加 · 您可能不知道關於 useState 的 7 ...· 5 月前 ·

I'am Running Spark Job that does hbase scan. However I get an error java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V

As I looked it up, it is caused by version mismatch between the hbase-client.jar to hbase version. However I used only hdp compiled jars.

My HDP version is 2.4.3.0

I run the sumbit the flowwing way:

export HADOOP_CONF_DIR=/etc/hadoop/conf/

export SPARK_CONF_DIR=/etc/spark/conf

/usr/hdp/current/spark-client/bin/spark-submit

--class MyClass

--master yarn-cluster

--num-executors 4

--driver-memory 1g

--executor-memory 4g

--executor-cores 6

--conf spark.driver.cores=6

--conf spark.storage.memoryFraction=0.8

--conf spark.shuffle.memoryFraction=0.1

--conf spark.yarn.jar=/usr/hdp/current/spark-client/lib/spark-hdp-assembly.jar

--conf spark.yarn.executor.memoryOverhead=2048

--conf spark.akka.frameSize=100

--conf spark.driver.extraJavaOptions="-Xss10m -XX:MaxPermSize=512M -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC "

--conf spark.executor.extraJavaOptions="-Xss10m -XX:MaxPermSize=512M -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC "

--jars /usr/hdp/current/hive-client/lib/hive-common.jar, /usr/hdp/current/hive-client/lib/hive-hbase-handler.jar, /usr/hdp/current/hbase-client/lib/hbase-common.jar, /usr/hdp/current/hbase-client/lib/hbase-server.jar, /usr/hdp/current/hbase-client/lib/hbase-client.jar, /usr/hdp/current/hbase-client/lib/hbase-procedure.jar, /usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar, /usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar, /usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar, /usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar, hdfs://mycluster:8020/lib/java/dependencies/mysql-connector-java-5.0.8-bin.jar

hdfs://mycluster:8020/lib/scala/myjar.jar

Terms & Conditions

Unsubscribe / Do Not Sell My Personal Information

Supported Browsers Policy

Apache Hadoop and associated open source project names are trademarks of the Apache Software Foundation. For a complete list of trademarks, click here.