I am pretty new to Spark and running it in local mode from eclipse on machine having configuration as Windows 10 with 8GB RAM.I was running pregel algorithm for summation of data at each nodes as per following link
Aggregation Summation at each node
I was running for Graph<Double,Double> for
1) One root node which in turn was having relationship of 1:1 with other 10 secondary node
2) These 10 secondary node was in turn having average of 1:6 with tertiary node and the total count of tertiary node is around 69.
I took around 8 minutes to run producing these two types of logs recurrently
12:55:38.242 |-INFO [Executor] Finished task 198.0 in stage 41.0 (TID 3012). 973 bytes result sent to driver
12:55:38.242 |-INFO [TaskSetManager] Starting task 199.0 in stage 41.0 (TID 3013, localhost, partition 199, PROCESS_LOCAL, 5553 bytes)
12:55:38.243 |-INFO [TaskSetManager] Finished task 198.0 in stage 41.0 (TID 3012) in 4 ms on localhost (199/200)
12:55:38.243 |-INFO [Executor] Running task 199.0 in stage 41.0 (TID 3013)
12:55:38.243 |-INFO [BlockManager] Found block rdd_93_199 locally
13:54:53.479 |-INFO [TaskSetManager] Finished task 76.0 in stage 11.0 (TID 69) in 29 ms on localhost (58/400)
13:54:53.480 |-INFO [Executor] Running task 78.0 in stage 11.0 (TID 70)
13:54:53.486 |-INFO [ShuffleBlockFetcherIterator] Getting 0 non-empty blocks out of 1 blocks
13:54:53.487 |-INFO [ShuffleBlockFetcherIterator] Started 0 remote fetches in 1 ms
13:54:53.495 |-INFO [Executor] Finished task 78.0 in stage 11.0 (TID 70). 3844 bytes result sent to driver
How to optimise it or the final option is to increase RAM?
