5

I have a use case where I have a streaming job running getting input data from kafka queue. And I have a reference data of 1 million rows which gets updated every hour. I load the reference data in the driver and then broadcast it to the workers. I would like to update this broadcast variable (in the driver) and resend it to workers.

What would be the best way to do this within spark, without introducing hbase/redis/cassandra etc?

And how reliable is this?

Do let me know if more information is needed. Thank you in advance. =)

Subba Rao
  • 165
  • 14

1 Answers1

3

Answer to the similar question was given later here: How can I update a broadcast variable in spark streaming?

In short, you will need to: "unpersist" broadcast variable, update, and rebroadcast it.

PS. formally this question is not a duplicate, because it was posted earlier.

Community
  • 1
  • 1
Timofey Chernousov
  • 1,284
  • 8
  • 12