1

I read the 2 GB limits in spark due to bytebuffer limitation of lnteger.MAX_INT. I understood that memory block size can't exceed due to this, so any record in an RDD partition should be less than 2gb.

But I read here(Why does Spark RDD partition has 2GB limit for HDFS?) an RDD partition itself can't exceed 2gb. And repartition into smaller partitions worked. Is it true? Please help me understand it..

Tom Sebastian
  • 3,373
  • 5
  • 29
  • 54
  • Is this a programming question or a question about server administration? (If its the latter, this isn't the right place to ask) – JeffUK Mar 03 '19 at 20:24
  • 2
    It is a programming doubt, we were facing issue with it application and trying to analyse reason – Tom Sebastian Mar 03 '19 at 23:12

0 Answers0