is there a way to reset spark Accumulator by the driver, so that executors Accumulator will be cleaned as well.
I have an Accumulator which is actually a List> all executors add tuples into the accumulator and the driver reads it. the problem is that executors heap gets filled.
When the driver reads all the data the executors does not need the local data.
thanks
I wonder if you have tried to reset an Accumulator by the driver using Accumulators' public method setValue().
You might be able to use this, i.e.
setValue()method, to reset accumulators’ values to smaller objects (e.g. the “zero” object of whatever your accumulator type is, or ‘null’ if you’re sure that the accumulator will never be accessed again).
