How does a process-based client share logical cores with other independent processes?

Question

My main thread A is running on logical core #1.

My local machine has 16 logical cores.

I create a process-based Client with 1 process.

By default the client is created with as many threads as the number of logical cores: 16.

from dask.distributed import Client

if __name__ == "__main__":
    print("Main thread A")
    with Client(processes=True, n_workers=1) as client:
        print(f"client spawned as an independent process: {client}")
--------------------
>>> Main thread A
>>> client spawned as an independent process: <Client: 'tcp://127.0.0.1:51522' processes=1 threads=16, memory=31.72 GiB>

1. How do I ensure that main thread A on logical core #1 never gets slowed down by the client?
2. More generally, how do the main thread and an independent process-based client share logical cores?

There is a partial answer here:

The OS takes care of waking threads and running them in the cores of your CPU

3. Would Client(processes=True, n_workers=1, threads_per_worker=15) ensure that the Client does not try to use logical core #1?

Answer 1

First let me clarify some points:

• Client constructor is a shortcut to running a LocalCluster, so a distributed Dask Cluster running on a single machine.
• This LocalCluster is composed of a Scheduler, running on your main process in its own event loop.
• It is also composed of at least one Dask Worker, running in an independent process.

How do I ensure that main thread A on logical core #1 never gets slowed down by the client?

It's hard to really do this. Dask will never pinned its internal Client, Scheduler or Workers to a single physical core. As you cited above, it's your OS job to spread threads and work on your physical cores.

How do I ensure that main thread A on logical core #1 never gets slowed down by the client?

Again, it's all about OS work here for managing resources.

Would Client(processes=True, n_workers=1, threads_per_worker=15) ensure that the Client does not try to use logical core #1?

No it won't. However, it will ensure that your Dask Worker will never use more than 15 threads. Witch should leave one thread (so the equivalent of one logical core of resources) free for your main thread or any other task. This assuming the tasks computed on your Dask cluster never uses more than one logical core.