1

I'm trying to use Dask local cluster to manage system wide memory usage,

from dask.distributed import Client, LocalCluster
cluster = LocalCluster(scheduler_port=5272, dashboard_address=5273,memory_limit='4GB')

I connect with:

client = Client('tcp://127.0.0.1:5272')

I have 8 cores and 32 GB. The local cluster distributes 4GB * 4 = 16GB memory (I have another task that required about 10GB memory) into the local cluster. However, previously there are some tasks I could finish well without calling client = Client('tcp://127.0.0.1:5272'). After I call client = Client('tcp://127.0.0.1:5272'), memory error triggered. What can i do in this scenario? Thanks!

I'm thinking if it is because each worker is only allocated 4GB memory... but if I assign memory_limit='16GB'. If it uses all of the resources it would take 64GB. I don't have that much memory. What can I do?

SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46
hg628193hg
  • 395
  • 3
  • 14

1 Answers1

1

It's not clear what you are trying to achieve, but your observation on memory is correct. If a worker is constrained by memory, then they won't be able to complete the task. What are ways out of this?

  • getting access to more resources, if you don't have access to additional hardware, then you can check coiled.io or look into the various dask cloud options

  • optimizing your code, perhaps some calculations could be done in smaller chunks, data could be compressed (e.g. categorical dtype) or there are other opportunities to reduce memory requirements (really depends on the functions, but let's say some internal calculation could be done at a smaller accuracy with fewer resources)

  • using all available resources with a non-distributed code (which would add some overhead to the resource requirements).

SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46
  • Thanks for your answer, so actually this specific task maybe require about 5GB. For the local cluster, I actually allocated 16GB total, although only 4GB per worker. Theoretically, it shall still complete the task... What if in the future, I have a machine that have 96 cores and 256GB, and I want to constraint "max_dask_usage" to be 200GB, then each worker only had 2GB memory....? Do you see any method i could use immediately to both constraint "max_dask_usage" and complete this task? Thanks so much for your help! – hg628193hg Aug 06 '21 at 16:05
  • Memory usage is tricky, you might need a memory profiler, e.g. for pandas it's recommended to have memory that is 4-5 times the size of the dataframe... – SultanOrazbayev Aug 06 '21 at 16:32
  • In my current scenario? is it possible to set max memory to be 16GB total and max core =8 while enabling the possibility of 1 worker taking all 16GB memory? – hg628193hg Aug 06 '21 at 17:35
  • In general yes, though I don't know if there is advantage in having one worker that uses all resources... this might be relevant: https://stackoverflow.com/a/49407253/10693596 – SultanOrazbayev Aug 06 '21 at 17:43