I am trying to run the following code using Spark in python:
def create_quantum_games(self, number_of_games):
"""
Creates given number of quantum games
"""
current_datetime = datetime.now().strftime("%Y%m%d%H%M%S")
file_path = self.base_file_path + current_datetime
machines_rdd = self.sc.parallelize(range(10), 10)
number_range = utils.get_number_list(self.precision)
payoffs_rdd = machines_rdd.map(lambda _: self.payoffs_func(number_range))
games_rdd = payoffs_rdd.map(lambda payoff: self.create_game(payoff, self.leaves))
games_rdd.saveAsTextFile(file_path)
self.sc.stop()
When running this code I always get the following runtime error:
RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
However, if I comment out the line games_rdd.saveAsTextFile(file_path) there is no such error.
Am I using saveAsTextFile wrong? What I want to achieve is, that the contents of the games_rdd get saved to a text file (line by line).
Is there a better approach to achieve this?
