1

Let's assume the configuration 14 executors, each with 10 GB memory and my input data size is 1000GB.

Q1 - How executors will process the huge data because data size is more and how partion logic works here ?

Q2 - How caching can help here for better performance and what caching strategy can be used ? (Mem only , Mem and disk etc )

prady
  • 563
  • 4
  • 9
  • 24

1 Answers1

2

1) Your 1000GB input would be split into blocks/chunks/partitions that are usually ~128MB each.

2) Caching should generally only be done if the data is computationally (or bandwidth) expensive, your cluster has enough memory, including overhead. You simply can't cache your entire 1000GB input with only 140GB total memory, but you may be able to cache other data sets for joins or an aggregate of the 1000GB input.

For more background information about #1, see these Q&As:

How are stages split into tasks in Spark?

How to split the input file in Apache Spark

How does the Apache Spark scheduler split files into tasks?

Garren S
  • 5,552
  • 3
  • 30
  • 45
  • Thanks for the response. Read the 3 above threads which explains about how data is split into tasks. Still there is one question which remains open, 1000 GB data can't fit in to 140 GB (14 Executors * 10 GB). So will the job takes longer run to do recursive operations to complete the job ? – prady Jan 29 '18 at 05:12
  • 1
    What does recursion have to do with reading in input larger than can be statically cached in your cluster? Spark, like many systems, works in batches, iteratively processing chunks of data – Garren S Jan 29 '18 at 17:17