0

It's a phenomenon about cpu intensive tasks when runing by python. I can not find a term to describe it.And English is not my mother tongue.

I have a program which is written by python. It is used to keep connection to the rabbitmq server,and do a task which is a cpu intensive task.

I use multithreading to run the cpu intensive task.and want to check the thread's status replace call method join.(for the reason:I should send heart beat by handle),just like the logic:

while True:
    if thread.is_alive(): # status check method
        time.sleep(5)
        log.debug("need continue")
        send_heartbeat()
    else:
        break

But it will lose rabbitmq's connection.

The rabbitmq loged: more than 60's not get the heartbeat from the connection,so close it .

And I tried to get more log:

  1. the log about "need continue" will not log out ,during the task(not all the time,just a part time of the task do) every 5 seconds.

It shows like,the child thread's method ,is_alive(),block more than 5 seconds(what's more,it must will block more than 60 seconds,so that the rabbitmq server will close the conneciton)

Or maybe,the main thread, not get cpu execution right: the cpu execution is occupy by the child-thread,which is a cpu intensive task.The occupy time more than 60 seconds?

  1. I tried child process,the similar phenomenon shows.

It also blocked at some where,from the log time,not every 5 seconds.For used the multi-process,I thought it is blocked at the status check method.

Finally,I use multi-progress varibales to avoid use the status check method.And it run OK now.

But why, both the thread and process will block when doing some cpu intensive tasks(not all cpu intensive tasks show this)?

In my plain thinking:

the multi-thread just could not use multi-core, not means, cpu will not switch from one thread to another thread ?

Or, in multi-progress,Why will also block when call the status check method? also can not understand.

Could anyone tell me ,which kinds of cpu intensive task will show the phenomenon above, And why?

Or what key words should I search to get the reason above?

I had searched key words:cpu intensive task(In chinese), for I did not find a very precise word,I do not get any similer explains.

And I asked my colleague,alose not get any answer.

Thanks.

lanhao945
  • 427
  • 5
  • 21
  • 2
    Might be helpful: https://stackoverflow.com/questions/20939299/does-python-support-multithreading-can-it-speed-up-execution-time ... `multithreading` module is more relevant for io-bound tasks, `multiprocessing` module is for cpu intensive tasks. – รยקคгรђשค Jul 13 '21 at 03:59
  • @รยקคгรђשค Maybe,it is not what I want to get. – lanhao945 Jul 13 '21 at 04:28
  • 1
    Python uses the gil to ensure that only one thread is run by the interpreter at a time. If you want concurrency you must use C to release the gil, or use multiprocessing – Mad Physicist Jul 13 '21 at 04:42
  • @MadPhysicist Sure, Python use gil to ensure that only one thread is run at a time. But why the child thread run in a time, which may be 60 seconds more?where is the cpu switch.Or multi-progress,where comes the block?can not use gil to explain... – lanhao945 Jul 13 '21 at 05:46
  • Someone told me, some infor belong to progress,When use multi-progress, if the process go into endless loop, it will lock the infor.So the other process check the status,will block util break out the endless loop. **Anyone can tell more about it ?** – lanhao945 Jul 13 '21 at 06:52
  • `is_alive` shouldn't block, it *never* did for me and the doc doesn't say it does. Make sure you don't have any other programming error somewhere else. Maybe 5 sec is too much of a pause? maybe the thread *stops* and the loop exit thereby stopping sending heartbeats. Python handles threads like any other programming language: by relying on the OS (which doesn't really care about process vs thread). Using a process could be useful (for python) only if you have *a lot* of shared states, to avoid locking on the GIL. Otherwise threads are fine. I did it hundreds of times and it worked as expected – Margaret Bloom Jul 17 '21 at 07:40
  • @MargaretBloom But it shows,just like blocked. It's sure that,the main thread's loop not exit,and the child thread not stop(from the log the the result). It just show,the loop,send heardbeat (between each time) more than 5 seconds some times.I asked others,told me,may be the child thread has any loop that not exit in a few times.such as Iterative solution？It makes the cpu not switch to the main thread. In the multi-process, also show just like blocked(almost the same performance). – lanhao945 Jul 19 '21 at 01:01
  • That's probably the symptom of a bug somewhere. The OS is *pre-emptive*, it doesn't care if a thread is using the CPU 100% of its quanta time or if it's using it only 0.01% of said time. The OS will always switch to another thread when it's time. You can test this with two simple threads: one prints something every second, the other just do some busy loop. You'll see a new line of output every second. If you can, somehow, make a [mcve] that would help to make this question answerable. – Margaret Bloom Jul 19 '21 at 07:12
  • @MargaretBloom Thanks very much.Let me some times to prepar a minimal reproducible example. – lanhao945 Jul 19 '21 at 08:15
  • @MargaretBloom Sorry, I tried two threads,one for log,another one for count.But not show block.It is hard for me to prepar a minimal example. I just call a third part api. the content is about counting points cloud,which has millions and tens of millions of points,make a large np data.And will occupy all most 100%cpu(It is hard for me to occupy so much just use threads). and occupy all most 50-60 G RAM.I do task about connecting to other serivce by rabbitmq rpc. – lanhao945 Jul 19 '21 at 08:46
  • It's hard to make a minimal example :) That's a very big workload, it may stress out your system to the point of slowing down everything. But I don't know, it's also hard to tell :) – Margaret Bloom Jul 19 '21 at 09:01
  • @MargaretBloom If I meet the same question and can show a minimal codes,Would I `@` you below the quesion? – lanhao945 Jul 19 '21 at 09:22
  • I'm not an expert on the subject. I was trying to help you write a good question because good questions get good and fast answer here ;) – Margaret Bloom Jul 19 '21 at 10:42

0 Answers0