Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

I submitted my code to the cluster to run, but I encountered the following error.

java.lang.IllegalArgumentException: Too large frame: 5211883372140375593 at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)

and my submit command is like this

spark-submit
--master spark://172.16.244.8:6066
--deploy-mode cluster
--num-executors 3
--executor-cores 8
--executor-memory 16g
--driver-memory 2g
--conf spark.default.parallelism=10
--class ParallelComputing
hdfs://172.16.244.5:9000/qlt/portrait-batch-0.0.1-jar-with-dependencies.jar

what is the reason

I've got the exception when I tried to use REST submission protocol against the regular 7077 port (not 6066 ). – Jacek Laskowski Aug 2, 2021 at 20:11 Hi, I'm using Kubernetes and facing this same issue. Can you please share more detailed answer – Atul Singh Rajpoot Sep 8, 2021 at 14:21

Thanks for contributing an answer to Stack Overflow!

• Please be sure to answer the question . Provide details and share your research!

But avoid …

• Asking for help, clarification, or responding to other answers.
• Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers .