1

Using python 3.8.8 on a windows box, how to determine the maximum number of processes that can be run per CPU core? I followed some previously asked related questions like 1,2,3,4.

Now the python docs on this subject suggest to use len(os.sched_getaffinity(0)). However, when I try in code, it returns an error,

import os
print(len(os.sched_getaffinity(0)))

AttributeError: module 'os' has no attribute 'sched_getaffinity'

Then I try another code snippet to identify the available functions,

import os
print(dir(os))

and it gives me a list of functions wherein sched_getaffinity() is not available.

My environment is:

  • Windows 10 64-bit
  • Python 3.8.8
  • 8 core processor
mnm
  • 1,962
  • 4
  • 19
  • 46
  • The only way to set processor affinity in windows 10 is manually, so cpu_count is as good as it gets. Check https://stackoverflow.com/questions/64189176/os-sched-getaffinity0-vs-os-cpu-count for further information about what you're looking for (basically you're on your own). – thebjorn Jun 28 '22 at 06:51
  • To me, it seems you either misread the docs or formulated the wrong question. `len(os.sched_getaffinity(0))` would give you the number of different _cores_ a process can run on, not the other way round. There is no restriction for the number of processes which can run per core. All you can do is take a specific process and pin it to specific cores. To do that in Windows you can use [psutil](https://psutil.readthedocs.io/en/latest/#psutil.Process.cpu_affinity). – Darkonaut Jun 28 '22 at 12:45

1 Answers1

3

The number of processes a CPU core can run simultaneously is equal to the number of threads per CPU core. And to get the number of threads per CPU core in python you can do something like this:

import psutil
total_threads = psutil.cpu_count()/psutil.cpu_count(logical=False)
print('You can run {} processes per CPU core simultaneously'.format(total_threads))

To check different processors your script can run on you can use:

print(psutil.Process().cpu_affinity())
Pythonista
  • 185
  • 12
  • when i run this line of code on my pc `print(psutil.Process().cpu_affinity())`, it gives 2 processes per CPU core. I have a 8-core CPU. Does this mean that I can execute only 16 processes at any given instance of time? – mnm Jul 02 '22 at 06:17
  • It means you can run 2 processes SIMULTANEOUSLY at any given instance of time – Pythonista Jul 02 '22 at 06:31
  • a follow up Q to your response.. suppose someone claims to execute >100 processes per CPU core of an 8 core processor? Is it possible or a bluff? – mnm Jul 02 '22 at 06:55
  • Yes CPUs execute a large number of processes. To check the number of processes your CPU is running press CTRL, SHIFT Esc on Windows and you will see the number of active threads and processes your CPU is running! (You can take threads as the child of processes) A process can have many threads PS: My 6 core processor is running 200 processes and 2314 threads at the moment – Pythonista Jul 02 '22 at 07:54
  • thanks for detailed response. I've a brief idea on processes and threads. My question is, if a 6 core processor can run 200 processes then why do we have technologies like **cloud computing AWS/GCP** in existence? After all service like AWS aids in scalability by executing innumerable processes at given time instance. Certainly there must be a bottleneck with **CPU based multiprocessing** that AWS has addressed? – mnm Jul 03 '22 at 01:31
  • Maybe! but services like cloud computing have other advantages like LOAD BALANCING; like if your website or web app is going through a peak time and your CPU is unable to manage it your are not going to buy another 8 core CPU for that and if you are on cloud you can simply use something called load balancer which will automatically add an additional server during peak time and will delete it automatically after the peak time is over! – Pythonista Jul 03 '22 at 10:02
  • Moreover in cloud you don't really need to manage the hardware and your website or web app will not go down just because a hard drive of your CPU got corrupt. – Pythonista Jul 03 '22 at 10:04
  • although the original question that I asked has largely remained unanswered, I'll still award you the bounty because, (1) your diligence and patience to answer at least partially the original question and related queries; (2) bounty period expiring & absence of canonical complete answer I was looking for. – mnm Jul 03 '22 at 23:56