7

I submitted my code to the cluster to run, but I encountered the following error.

''' java.lang.IllegalArgumentException: Too large frame: 5211883372140375593 at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)

'''

and my submit command is like this

spark-submit
--master spark://172.16.244.8:6066
--deploy-mode cluster
--num-executors 3
--executor-cores 8
--executor-memory 16g
--driver-memory 2g
--conf spark.default.parallelism=10
--class ParallelComputing
hdfs://172.16.244.5:9000/qlt/portrait-batch-0.0.1-jar-with-dependencies.jar

what is the reason

zhicheng
  • 156
  • 1
  • 6

2 Answers2

4

The reason is because the version of spark does not match the version of the cluster, which can be solved by modifying the local spark version to be consistent with the cluster.

zhicheng
  • 156
  • 1
  • 6
0

In my spark project I face this same issue ,I found my spark version is 3.3.1 and hadoop version(hadoop-client-runtime) is 3.3.2. So I change my hadoop-client-runtime version from 3.3.2 to 3.3.1. After I tested, the problem was solved.

Ke Liu
  • 1
  • 1