Dask Memory Management with Default Scheduler

Question

I have been trying to manage the memory usage of Dask on a single local machine. For some reason, the default Dask Client() and LocalCluster() scheduler always seem to break, however Dask works great without specifying the scheduler and thus the default scheduler works the best for my purposes, however I am finding almost no documentation on this default scheduler let alone how to set a RAM limit on it. All of the information is for their specialized distributed client which does not seem to work. Does anyone know what/where the default scheduler documentation can be found or what they are even using as the default scheduler (as I cannot even find that information on their site/documentation). Or if not, has anyone found a solution to limit the RAM usage without using Client()?

Cheers

What do you mean by default scheduler? Maybe, you can add code examples. — Vladyslav Moisieienkov, Nov 10 '18 at 20:42
The default scheduler is the scheduler which is used by default... Aka no specified scheduler or even reference to a scheduler. All i use is import dask. I could copy error codes, but they generate 100s of lines of errors. — WolVes, Nov 11 '18 at 21:10

score 2 · Answer 1 · answered Dec 23 '18 at 21:58

2

in order to control the memory-limit in my local dask cluster i use the following set up:

ncores = 2
cluster = LocalCluster(n_workers=ncores, threads_per_worker=1,memory_limit =8e9)
client=Client(cluster)

answered Dec 23 '18 at 21:58

s1mc0d3

523
2
15

Dask Memory Management with Default Scheduler

1 Answers1

Linked