0

I'm new on apace spark,how we can create spark context for web based applicaiton.

The below i've used,

from pyspark.context import SparkContext
from pyspark.conf import SparkConf
conf = SparkConf().setAppName('Test App').setMaster('spark://localhost:7077')
app = Flask(__name__)

@app.route('/find_possible_events', methods=['POST'])
def find_possible_events():
     sc=SparkContext.getOrCreate(conf=conf)
     data=sc.parallelize([20,30,40,50])
     d=data.collect()
     sc.stop
     return d

The above works fine when we had single at same time, but if we had multiple request means got error.

Exception in thread "dispatcher-event-loop-8" java.lang.OutOfMemoryError: GC overhead limit exceeded

when i took at the spark worker log, it says spark context is not avaliable.

Thanks in advance

Alper t. Turker
  • 34,230
  • 9
  • 83
  • 115
Robert
  • 3,373
  • 1
  • 18
  • 34
  • @RameshMaharjan Thanks for reply. so we wont close spark context ?? – Robert Jun 26 '18 at 05:42
  • Without going into details embedding like this is not the way to go, and a good solution is to broad for SO format. I recommend starting with [Best Practice to launch Spark Applications via Web Application?](https://stackoverflow.com/q/40300222/8371915). – Alper t. Turker Jun 26 '18 at 10:06

0 Answers0