1

As per title, if I am creating workers via helm or kubernetes, is it possible to assign "worker resources" (https://distributed.readthedocs.io/en/latest/resources.html#worker-resources) after workers have been created?

The use case is tasks that hit a database, I would like to limit the amount of processes able to hit the database in a given run, without limiting the total size of the cluster.

limx0
  • 57
  • 6
  • Okay another dig through the code, I found `set_resources`. I assume I could do a `client.run(lambda x: set_resources('my-db')`. Would this be the most appropriate way to achieve this? – limx0 Apr 09 '19 at 23:52

1 Answers1

0

As of 2019-04-09 there is no standard way to do this. You've found the Worker.set_resources method, which is reasonable to use. Eventually I would also expect Worker plugins to handle this, but they aren't implemented.

For your application of controlling access to a database, it sounds like what you're really after is a semaphore. You might help build one (it's actually decently straightforward given the current Lock implementation), or you could use a Dask Queue to simulate one.

MRocklin
  • 55,641
  • 23
  • 163
  • 235