Why executor memory used is shown greater than total available memory on spark web UI?

Asked May 13 '20 at 07:30

Active May 13 '20 at 14:36

Viewed 367 times

I have a spark structured streaming job that is running for around last 3 weeks. When I open the Executors tab on spark web UI, it shows

 memory used - 36.1GB
 total available memory for storage - 3.2GB

For this application executor memory is set to 6GB.

Please refer to the below screenshot of Spark UI

I am not sure how should I interpret this information.

edited May 13 '20 at 14:36

asked May 13 '20 at 07:30

Ravi

for Spark Explanation: `The Storage Memory column shows the amount of memory used and reserved for caching data.` [link](https://spark.apache.org/docs/3.0.0-preview/web-ui.html#executors-tab) I was also confused why `used memory` greated than `reserved memory` – roamer Sep 25 '20 at 03:46

0 Answers0