3

I am little confused with working mechanism of spark for in-memory computation. If spark does the processing in-memory , how does it compute 100 TB of data when it has 16 GB of in-memory storage

thebluephantom
  • 16,458
  • 8
  • 40
  • 83
Learnerrr
  • 127
  • 1
  • 5
  • Check [this](https://www.youtube.com/watch?v=7ooZ4S7Ay6Y) video probably it will answer many questions that you might have. – abiratsis Mar 02 '19 at 15:51

1 Answers1

1

Spark will work on blocks of data that fit within the resources and computing capabilities of the machine. This work will be split in several iterations during which spark will load data to memory, process it and write the result back to disk if necessary to work on the next chunk of data.

Hichem BOUSSETTA
  • 1,791
  • 1
  • 21
  • 27